CPSCore adds pre-commit transaction hooks support on ZODB 3.2

Issue :

We recently realized on a CPS instance while plugging an external catalog (ZSQLCatalog actually) that a *lot* of reindexations (something like 10) occured at creation time. Same behavior for content modifications and still the same with workflow transition excecutions.

Usually, the problem is not visible but as soon as you get big catalog (for instance 600 K entries) then the creation or workflow transition execution had a tendency to be slow down.

First step : object repository objects

Historically, the object repostiory objects within CPS3 were indexed. It wasn't necesarly anymore since CPS branch 3.1.x and the new rearch mecanism that basically aggregates the repository object attributs on the proxy itself with the help of dedicated proxy indexes. (Ben did a great work at this time and the search speed of CPS grew up big time after this).

Thus, the first step was to avoid repository object indexations.

Direct benefits :

• Half of the indexations avoided
• Lighter catalog and thus more efficient ones when they become really big.

Second step : getting only one reindexation per-object and per-transaction no matter what appends to the objects during the transaction.

First, I've been trying to figure out where to call my reindexObject() method within the CPS architecture. It's definitely not mangeable especially when you got during the same transitions : object reindexation, object security reindexation and workflow transition executed with the object involved. The CMF is definitely not optimized at all on this point.

Then, we started to discuss with Florent about a possible post-publishing hook on the Zope publisher. No existing hooks were available for this and no way to patch properly the framework for our extensions.

We discussed about the problem on zope-dev and zodb-dev (thanks to the people over there for the answers)

The problem we were faced to was that the ZODB-3.2 (the one Zope-2.7.x is using) doesn't provide good ways to hook just *before* the first commit phase of the transaction.

There's a way to register a hook executed at the beginning of the first commit phase implementing this interface :

    http://cvs.nuxeo.org/cgi-bin/viewcvs.cgi/CPS3/CPSCore/interfaces/IZODBTransactionHook.py?rev=1.1&view=auto

and registering the object implementing this interface like this  :

>>> get_transaction().register(hookInstance)

The problem is that this hook is called too late in the commit process (during phase1) and the objects are already frozen at this time. It's thus not possible to deal with Persistent Objects and thus no way to cope with our Indexation problems.

We discovered that Zope-3.3.x would have provide a way to do the job with the Synchronizers and the beforeCompletions() API :

See the discussion there :

http://mail.zope.org/pipermail/zope-dev/2005-April/024566.html

I finally decided to extend the ZODB-3.2 transactions to make them support pre-commit hooks with a subscriber mecanism that I may use for an Indexation manager subscriber.

A pre transaction commit hook has been added to the ZODB Transaction supporting subscribers registration. Each subscriber has a commit() method called just *before* the first commit phase that contains the actual code to execute and an abort() method called at the beginning of the abort() method of the transaction class. :

Pre-commit hook definition with subscribers :

http://cvs.nuxeo.org/cgi-bin/viewcvs.cgi/CPS3/CPSCore/PatchZODB.py

Subscriber definition :

http://cvs.nuxeo.org/cgi-bin/viewcvs.cgi/CPS3/CPSCore/TransactionCommitSubscribers.py?rev=1.1&view=auto

Subscriber interface :

http://cvs.nuxeo.org/cgi-bin/viewcvs.cgi/CPS3/CPSCore/interfaces/ITransactionCommitSubscriber.py?rev=1.1&view=auto

You may register your own subscribers by implementing the interface

above and register them to the list of pre-commit transaction subscribers.

This is checked in and available in the HEAD of CPSCore.

Note that the extensions currently defined within CPSCore will disapear.

The threads are still running talking about an implementation for ZODB-3.4 of a simple before transaction commit hook :

• http://mail.zope.org/pipermail/zope-dev/2005-April/024566.html
• http://mail.zope.org/pipermail/zodb-dev/2005-April/008661.html
• http://mail.zope.org/pipermail/zodb-dev/2005-April/008667.html
  

   J.