Florent Guillaume

JavaOne 2007 Slides

JavaOne is now finished. We had a very interesting few days there, meeting lots of people interested in Nuxeo and wanting to know more about the technology and the business around it. My talk itself was well received, and I hope it gave people a good sense of what is possible and desirable in the Open Source ECM problem space.

You can view the slides of the talk here.

I also attended some very interesting sessions regarding subjects that are important when writing an ECM framework: persistence, distribution, scalability, validation, object-relational modeling, etc. A number of efforts are ongoing to standardize as JSRs some important aspects related to this. In the coming days I'll be blogging in more detail about these JSRs that we'll be tracking closely.

For our first participation, this was a very fruitful experience, and we'll certainly be going again next year!

Discover Nuxeo 5 at JavaOne 2007

JavaOne 2007 is in a few days now, and we'll be giving a talk about Nuxeo 5.

The talk has ID TS-4532 (in the Java EE track), and takes place on Thursday the 10th at 5:30 pm in Hall E - 134. The title of the session is Building an Embeddable Enterprise Content Management Core with the Latest Java Technologies.

This is a great opportunity if you want to discover more about how Nuxeo 5 is architected, see demos of the product, or talk to us about technical or business-related questions.

As a teaser, here's the overview of the talk:

Learn about the design and use of Nuxeo 5, an embeddable, extensible Enterprise Content Management framework for Java EE and other platforms

Agenda:

What is Enterprise Content Management
What do we want to achieve
Core Framework
Core and High-Level Services
UI Layers
Extensible and Pluggable
Using the Framework

See you in San Francisco!

What's the point of JCR?

Nuxeo is switching its ECM to Java, and we're using JCR for our document storage. JCR (Java Content Repository, standardized by JSR-170 and the upcoming JSR-283) is a young specification with a promising future — but what's its point, you may ask, as all existing content management systems are already storing content very well without it? Its goal is interoperability between vendors, which will make it possible for people who write applications needing to store content to have a unified API for such manipulations. All major content repository vendors are active in the JSR-283 expert group, and all are working on JCR bindings for their various proprietary repositories.

Of course a standardized and wildly successful way of manipulating content already existed before JCR: SQL. But SQL and JCR have a different focus:

SQL is a language; it manipulates rows and is geared toward generic relation manipulation,
JCR is a Java API; it manipulates nodes and is geared toward hierarchical manipulation (parent-children).

SQL and JCR have quite different underlying data models:

SQL's model is that of tables with fixed schemas, and relations between tables,
JCR's model is that of a tree of nodes with flexible schemas and with parent-children relations as the main focus — although other types of relations exist.

JCR also offers higher level features than SQL, notably workspace and version management.

For many kinds of applications, there is a focus on being able to arrange documents in folder hierarchies, and to have a wild variety of structure for these documents. In this case, a storage model based on JCR is much more suited than something based on SQL.

It should be noted that many things in the computing world already are based on the notion of folder hierarchies storing arbitrary documents:

filesystems are a tree of (unstructured) documents,
Revision control systems are based on filesystem concepts but add a lot of structure on top of them,
WebDAV (along with DAV and DELTAV) is a protocol that addresses documents using a path, and where documents have a flexible set of properties,
most proprietary content management systems are based on (or offer the notion of) organizing the content in a hierarchy,
Nuxeo's CPS is itself based on this classic model.

This is why JCR has emerged as a common and useful storage API for all these use cases.

For an ECM framework like Nuxeo 5, JCR interoperability can be seen from two different directions:

JCR provides an API that we can use to store our content, to make us vendor-independent and flexible regarding storage,
JCR provides an API through which we can expose our content, which makes our platform usable by any external system that understands it — we're ourselves a vendor providing JCR bindings.

For its initial release, Nuxeo 5 is focusing on the use of JCR as its main storage implementation and uses Jackrabbit to store most of our content. In the future, we'll also provide JCR bindings so that the high-level content we provide can be directly accessed from external applications using JCR too.

Nuxeo 5: Unifying the content APIs

In a content management system, the actual data that the system or the users manipulate comes from many kinds of sources. Content can come from a JCR repository, or from a relational database, or from an LDAP directory, or from a semantic storage engine like Jena, or from any other kind of open or proprietary storage engine.

But fundamentally all these kinds of content, which I'll call "records", aren't very different:

a record can be created, viewed, modified, deleted,
a record can often be copied or moved,
a record obeys a schema that can be known to the system, this means that its individual fields are strictly typed,
when being viewed or modified, a record has a user interface that is based on forms, labels, widgets, depending on the schema,
records can be searched and a result set returned,
records can be listed in a compact form (search results, folder contents, user dashboard, workflow workitems, RDB table listing, user information browsing, etc.),
records have an identity (like a unique id) or a location (like a path), sometimes both.

One of the strengths of CPS is to use a common abstraction for many of these concepts, embodied in the CPSSchemas component. In Nuxeo 5 we want to go further and provide even more integration for all these, the base components for these abstractions are NXCore and NXTypeManager.

The reasons to strive for convergence are numerous:

this merges into unique concepts things that had been previously separated because of different implementation choices. For instance, an LDAP schema is not fundamentally different from an SQL schema (or from an XML Schema if one is interested in the relevant subset).
this gives the programmers a common API for all data-related operations, which means more reusability. For instance changing an attribute in an LDAP entry doesn't have to be different from changing the title of a document or changing a value in an RDB row; processing a list of search results to display them in a table doesn't have to be different from processing the children of a folder to display the folder's contents.
this gives the framework developers a way to optimize some operations because of commonalities in the underlying implementations. For instance you don't need three kinds of events dispatching for "LDAP entry modified", "RDB row modified" or "document modified".
this gives the users a unified way of manipulating different kinds of data when there's really no need to have a different UI for them. When a user fills a form it's really the same process whether he's modifying his personal preferences, adding a keyword to a document, or changing a quantity in an RDB row.
this allows very simple migrations between storage technologies, when these are felt necessary. A customer could start with an LDAP database for its user base and later have the need to move them to an RDB table. User entries in an RDB table may need to be versioned and moved to a JCR storage. An application should survive all these with only configuration changes and no code to rewrite.

It should be noted that this means that JCR is in no way the primary storage model for Nuxeo 5, it's only the first one to be implemented. In the future, it will be possible to store documents in LDAP or an RDB. When a suitable storage model is devised and implemented, you'll be able to apply workflow or versioning to RDB-based documents for instance.

This convergence is quite exciting to us, and our goal is to allow people to build complex applications with Nuxeo 5 in a more straightforward manner.

GenericSetup for CPS, CMF and Zope

GenericSetup is a framework to describe the configuration of a Zope site as a set of XML files (and sometimes other associated files). It can import profiles, which may create objects or change their configuration, and export profiles, which makes a snapshot of the configuration and writes it to a set of XML files.

GenericSetup provides a tool that can store snapshots of a configuration in the ZODB itself, where it can be examined and even modified. It can also do diffs between two snapshots, which is very useful to find out what changed in a configuration (it's a good idea to take a full snapshot anytime some significant changes are made to the configuration).

GenericSetup differentiates between Base and Extension profiles. A Base profile is a profile that describes "everything". When it is imported, it removes and overwrites any previous configuration. An Extension profile is a profile designed to be incrementally added on top of a previously existing configuration. It may of course overwrite some settings, or even in some case remove objects, but its goal is generally to add optional configuration on top of a main one

GenericSetup is based on a small number of concepts: the toolset, and some import and export steps. GenericSetup provides a framework where import and export steps can be written simply using Zope 3 adapters.

GenericSetup was born as "CMFSetup" but was later made generic and can be used by any Zope 2 application. It is now available at http://svn.zope.org/GenericSetup/trunk/. A subclass with additional features is used in CPS.

Setup tool

The setup tool is called setup_tool by default, and portal_setup in CPS (if missing, it can be instantiated by selecting "CPS Tools" from the ZMI Add menu).

At a given time, the setup tool knows about a few things:

the currently selected profile, which may point to an area in the filesystem where a profile has been registered, or to a path in the ZODB where a snapshot was taken,
the current toolset,
the current import steps,
the current export steps.

When a new profile it selected, its toolset, import steps and export steps XML files are loaded and merged with the tool's current ones.

Any import or export is based on the full toolset or import/export steps (even if the currently selected profile is an extension profile), but the source of XML configuration files depends solely on the currently selected profile.

Toolset

The file toolset.xml describes the Toolset, which is the set of tools needed in a given configuration. (While the name "tool" would suggest CMF, it's just a set of objects that can be instantiated at the root of the configured site.)

Beyond being based on data that is updated when an extension profile is selected, the toolset is a normal import/export step.

When the toolset is imported, all the objects in the toolset are instantiated if they're missing or if their class doesn't match with what it's supposed to be.

An excerpt of toolset.xml for the CPS Tree Tool would be:

<?xml version="1.0"?>
<tool-setup>
  ...
  <required tool_id="portal_trees"
            class="Products.CPSCore.TreeTool.TreeTool"/>
  ...
</tool-setup>

Import steps

Import steps (import_steps.xml) describe a set of configuration steps available for import, and their dependencies. A step is just the dotted name of a function, and the dependencies is simply the steps that have to be run before this one can be done (most steps depend on the toolset, because they need a base tool to be instantiated before it can be configured).

While an import step can do anything it likes, GenericSetup provides a framework based on Zope 3 adapters to simply describe how a single object is imported from XML, and to recurse among objects to create or configure them one by one.

Purge

During import, there are two possible behaviors, corresponding to the two kinds of profiles. For a Base profile, the import happens in "purge" mode, while for an extension profile the import doesn't purge. The functions and adapters doing the import have to take that into account when they read a profile.

In purge mode, every previous configuration has to be removed, so that the result is indistinguishable from an install from scratch. In non-purge mode, care must be taken to not overwrite or remove settings or objects which are not explicitely specified in the imported XML file.

An excerpt of import_steps.xml for the CPS Tree Tool would be:

<?xml version="1.0"?>
<import-steps>
  ...
  <import-step id="trees" version="20051230-01"
               handler="Products.CPSCore.exportimport.importTreeTool"
               title="Tree Tool">
    <dependency step="toolset"/>
    Import tree tool and tree caches.
  </import-step>
  ...
</import-steps>

Export steps

Export steps (export_steps.xml) describe the list of steps available for export. An export step is used in exactly the same way than an import step, except for the fact that there are no dependencies between export steps, and that instead of being read, XML files are written.

One important thing to note is that export steps cannot do the "incremental exports" that many people expect. When an extension profile is read by the import steps, only the available XML files for that profile are read. However when writing, there's no way to choose which XML files are relevant, so the whole profile for that step is written (and recursion is done in all subobjects).

It's possible that future versions of GenericSetup will have some capabilities to do incremental exports, but this is not possible for now.

An excerpt of export_steps.xml for the CPS Tree Tool would be:

<?xml version="1.0"?>
<export-steps>
  ...
  <export-step id="trees"
               handler="Products.CPSCore.exportimport.exportTreeTool"
               title="Tree Tool">
    Export tree tool and tree caches.
  </export-step>
  ...
</export-steps>

Adapters

The standard work done in an import or export step is to find the base tool, and call importObjects or exportObjects on it; these are recursive functions that take each object, find an adapter for them describing how the XML import or export is done, and call it.

For the CPS Tree Tool, the import steps and export steps call the following functions:

def exportTreeTool(context):
    """Export Tree tool and tree caches as a set of XML files.
    """
    site = context.getSite()
    tool = getToolByName(site, 'portal_trees', None)
    if tool is None:
        logger = context.getLogger('trees')
        logger.info("Nothing to export.")
        return
    exportObjects(tool, '', context)

def importTreeTool(context):
    """Import Tree tool and tree caches as a set of XML files.
    """
    site = context.getSite()
    tool = getToolByName(site, 'portal_trees')
    importObjects(tool, '', context)

This is pretty boileplate and could even be simplified in the future through ZCML declarations. Above, '' simply refers to the root of the profile directory.

The adapters are multi-adapters, adapting both an object and an import context (called "environ" in GenericSetup), to the IBody interface that basically describes a file body. They can be registered through ZCML using the standard statement:

<adapter
    factory=".exportimport.TreeToolXMLAdapter"
    provides="Products.GenericSetup.interfaces.IBody"
    for=".interfaces.ITreeTool
         Products.GenericSetup.interfaces.ISetupEnviron"
    />

This assumes of course that the exported object is described through an interface, for instance here declared as:

from zope.interface import Interface

class ITreeTool(Interface):
    """Tree Tool.
    """

The interface is implemented by the object's class with:

from zope.interface import implements

class TreeTool(UniqueObject, Folder):
    """Tree Tool that caches information about the site's hierarchies.
    """
    implements(ITreeTool)

    id = 'portal_trees'
    meta_type = 'CPS Tree Tool'
    ...

When doing an export, the adapters build a DOM tree for the configuration. When doing an import, they read the DOM tree and create or modify properties as needed. The standard adapters don't create subobjects or recurse in them, this is left to the importObjects function.

These adapters can be written easily because GenericSetup provides helpers for the common cases of objects configured only through standard Zope 2 properties (PropertyManager), or having subobjects (ObjectManager).

Of course all this can be changed for specific cases. Many older CMF tools are configured through things that are not standard properties, and for instance CPS needs to do recursion into more than simple ObjectManager subobjects. It is also possible to read and write files that are not XML files, CPS does this for the images included in portlet objects, where it writes a real image file.

Additional CPS feature: upgrades

CPS has extended the setup tool to provide a basic Upgrade feature, that is related to the configuration of the site but cannot be expressed by the standard GenericSetup profiles.

An upgrade step is registered through ZCML with something like:

<cps:upgradeStep
    title="Replace CMF URL tool with a CPS-specific one"
    source="3.3.4" destination="3.3.5"
    handler=".upgrade.upgradeURLTool"
    checker=".upgrade.check_upgradeURLTool"
    />

This describes between what CPS versions this upgrade is needed, how to do it, and how to check if it's already been done.

The setup tool lists which steps have not yet been done, and provides a way to run them one by one or all at once. At a given time, a CPS site "knows" (through a site-global property last_upgraded_version) up to which version it's been upgraded.

The checker is optional; if it would be too costly to check for the conversion, it can be omitted. This is the case typically for an upgrade step that would recurse in the content object to do some fixups; to check if they've already been done it would have to recurse in the content objects too, and that's too much work for a simple check. The setup tool won't list a step without a checker if the portal has already been upgraded to a later version than the step's destination version.

Object event dispatching in Zope

Here are some explanations about what happens in Zope 3.2 (and Zope 2.9 when using Five) when an event notification is sent by some code, up to a specific subscriber. It focuses more specifically on object events, which go through some additional hoops. All this is complex because there are many simple components that are linked together.

Let's start with some framework code that sends an event after an object has been added (similar to what zope.app.container.contained actually does):

event = ObjectAddedEvent(ob, container, name)
zope.event.notify(event)

In zope.event we have the definition for this function:

subscribers = [] # registered subscribers
def notify(event):
    for subscriber in subscribers:
        subscriber(event)

During initialization, zope.app.event.dispatching has registered a subscriber:

def dispatch(*event):
    # Iterating over subscribers assures they get executed.
    for ignored in zope.component.subscribers(event, None):
        pass
zope.event.subscribers.append(dispatch)

The function zope.component.subscribers will then call all matching subscribers.

During initialization, zope/app/event/configure.zcml has registered a subscriber for zope.app.event.interfaces.IObjectEvent with the handler zope.app.event.objectevent.objectEventNotify:

<subscriber
    for="zope.app.event.interfaces.IObjectEvent"
    handler="zope.app.event.objectevent.objectEventNotify"
    />

This handler does:

def objectEventNotify(event):
    adapters = zope.component.subscribers((event.object, event), None)
    for adapter in adapters:
        pass # Getting them does the work.

This means that the event will be redispatched, but this time a subscriber can match using multi-adaptation on both the object and the event interfaces, which gives much more flexibility and filtering possibilities.

During initialization, zope/app/container/configure.zcml has registered a multi-subscriber:

<subscriber
    for="zope.app.location.interfaces.ILocation
         zope.app.container.interfaces.IObjectMovedEvent"
    handler="zope.app.container.contained.dispatchToSublocations"
    />

Note that IContained, a base interface for most content objects, derives from ILocation, so this subscriber will match most content objects. Also, the IObjectAddedEvent sent initially derives from IObjectMovedEvent so it will be matched.

When using Five, in Five/event.zcml some similar subscribers are registered to react on IObjectManager instead of ILocation, and to dispatch using the same handler.

The dispatchToSublocations handler is:

def dispatchToSublocations(object, event):
    subs = ISublocations(object, None)
    if subs is not None:
        for sub in subs.sublocations():
            for ignored in zapi.subscribers((sub, event), None):
                pass # They do work in the adapter fetch

This redispatches the same event to all subobjects, where "subobjects" is defined using the ISublocations adapter. Now, zope/app/container/configure.zcml has an adapter:

<adapter
    for="zope.app.container.interfaces.IReadContainer"
    provides="zope.app.location.interfaces.ISublocations"
    factory="zope.app.container.contained.ContainerSublocations"
    />

Most Zope 3 container objects are also IReadContainer. The ContainerSublocations handler does simply get the sublocations using:

class ContainerSublocations(object):
    def __init__(self, container):
        self.container = container
    def sublocations(self):
        container = self.container
        for key in container:
            yield container[key]

When using Five, a similar adapter is registered:

<adapter
    for="OFS.interfaces.IObjectManager"
    provides="zope.app.location.interfaces.ISublocations"
    factory="OFS.subscribers.ObjectManagerSublocations"
    />

And the Five adapter does something comparable to the Zope 3 one:

class ObjectManagerSublocations(object):
    def __init__(self, container):
        self.container = container
    def sublocations(self):
        for ob in self.container.objectValues():
            yield ob

All in all, our initial event will be redispatched to the sublocations of the original object, and the process will be done recursively to all the sublocations.

Now some user code's configure.zcml can registered a multi-subscriber:

<subscriber
    for=".interfaces.IFoo
         zope.app.container.interfaces.IObjectAddedEvent"
    handler=".foo.reactOnAdd"
    />

Which ties to the handler:

def reactOnAdd(ob, event):
    """Does something."""

In the handler, which will be called for all sublocations, ob is any sublocation of the original object (including the object itself), and event is the original event (which means that event.object is the original object).

This concludes our dive into through Zope 3 events. You can read in http://blogs.nuxeo.com/sections/blogs/florent_guillaume/2005_11_08_events-in-zope-2-9 how object events can be used in Zope 2.9 to do what used to be done using manage_afterAdd.

Events in Zope 2.9

Zope 2.9 (and Zope 2.8 when using Five 1.2) introduces a big change: Zope 3 style container events.

With container events, you finally have the ability to react to things happening to objects without have to subclass manage_afterAdd, manage_beforeDelete or manage_afterClone. Instead, you just have to register a subscriber for the appropriate event, for instance IObjectAddedEvent, and make it do the work.

Indeed, the old methods like manage_afterAdd are now deprecated, you shouldn't use them anymore.

Let's see how to migrate your products.

Old product

Suppose that in an old product you have code that needs to register through a central tool whenever a document is created. Or it could be indexing itself. Or it could initialize an attribute according to its current path. Code like:

class CoolDocument(...):
    ...
    def manage_afterAdd(self, item, container):
        self.mangled_path = mangle('/'.join(self.getPhysicalPath()))
        getToolByName(self, 'portal_cool').registerCool(self)
        super(CoolDocument, self).manage_afterAdd(item, container)

    def manage_afterClone(self, item):
        self.mangled_path = mangle('/'.join(self.getPhysicalPath()))
        getToolByName(self, 'portal_cool').registerCool(self)
        super(CoolDocument, self).manage_afterClone(item)

    def manage_beforeDelete(self, item, container):
        super(CoolDocument, self).manage_beforeDelete(item, container)
        getToolByName(self, 'portal_cool').unregisterCool(self)

This would be the best practice in Zope 2.8. Note the use of super() to call the base class, which is often omitted because people "know" that SimpleItem for instance doesn't do anything in these methods.

If you run this code in Zope 2.9, you will get deprecation warnings, telling you that:

Calling Products.CoolProduct.CoolDocument.CoolDocument.manage_afterAdd
is deprecated when using Five, instead use event subscribers or mark
the class with <five:deprecatedManageAddDelete/>

Using five:deprecatedManageAddDelete

The simplest thing you can do to deal with the deprecation warnings, and have correct behavior, is to add in your products a configure.zcml file containing:

<configure
    xmlns="http://namespaces.zope.org/zope"
    xmlns:five="http://namespaces.zope.org/five">

  <five:deprecatedManageAddDelete
      class="Products.CoolProduct.CoolDocument.CoolDocument"/>

</configure>

This tells Zope that you acknowledge that your class contains deprecated methods, and ask it to still call them in the proper manner. So Zope will be sending events when an object is added, for instance, and in addition call your old manage_afterAdd method.

One subtlety here is that you may have to modify you methods to just do their work, and not call their super class. This is necessary because proper events are already dispatched to all relevant classes, and the work of the super class will be done trough events, you must not redo it by hand. If you call the super class, you will get a warning, saying for instance:

CoolDocument.manage_afterAdd is deprecated and will be removed in
Zope 2.11, you should use an IObjectAddedEvent subscriber instead.

The fact that you must "just do your work" is especially important for the rare cases where people subclass the manage_afterAdd of object managers like folders, and decided to reimplement recursion into the children themselves. If you do that, then there will be two recursions going on in parallel, the one done by events, and the one done by your code. This would be bad.

Using subscribers

In the long run, and before Zope 2.11 where manage_afterAdd and friends will be removed, you will want to use proper subscribers.

First, you'll have to write a subscriber that "does the work", for instance:

def addedCoolDocument(ob, event):
    """A Cool Document was added to a container."""
    self.mangled_path = mangle('/'.join(self.getPhysicalPath()))

Note that we're not calling the portal_cool tool anymore, because presumably this tool will also be modified to do its work through events, and will have a similar subscriber doing the necessary registerCool. Note also that here we don't care about the event, but in more complex cases we would.

Now we have to register our subscriber for our object. To do that, we need to "mark" our object through an interface. We can define in our product's interfaces.py:

from zope.interface import Interface, Attribute

class ICoolDocument(Interface):
    """Cool Document."""
    mangled_path = Attribute("Our mangled path.")
    ...

Then the class CoolDocument is marked with this interface:

from zope.interface import implements
from Products.CoolProduct.interfaces import ICoolDocument
class CoolDocument(...):
    implements(ICoolDocument)
    ...

Finally we must link the event and the interface to the subscriber using zcml, so in configure.zcml we'll add:

...
  <subscriber
      for="Products.CoolProduct.interfaces.ICoolDocument
           zope.app.container.interfaces.IObjectAddedEvent"
      handler="Products.CoolProduct.CoolDocument.addedCoolDocument"
      />
...

And that's it, everything is plugged. Note that IObjectAddedEvent takes care of both manage_afterAdd and manage_afterClone, as it's sent whenever a new object is placed into a container. However this won't take care of moves and renamings, we'll see below how to do that.

Event dispatching

When an IObjectEvent (from which all the events we're talking here derive) is initially sent, it concerns one object. For instance, a specific object is removed. The event.object attribute is this object.

To be able to know about removals, we could just subscribe to the appropriate event using a standard event subscriber. In that case, we'd have to filter "by hand" to check if the object removed is of the type we're interested in, which would be a chore. In addition, any subobjects of the removed object wouldn't know what happens to them, and for instance they wouldn't have any way of doing some cleanup before they disappear.

To solve these two problems, Zope 3 has an additional mechanism by which any IObjectEvent is redispatched using multi-adapters of the form (ob, event), so that a subscriber can be specific about the type of object it's interested in. Furthermore, this is done recursively for all sublocations ob of the initial object. The event won't change though, and event.object will still be the original object for which the event was initially sent (this corresponds to self and item in the manage_afterAdd method -- self is ob, and item is event.object).

Understanding the hierarchy of events is important to see how to subscribe to them.

IObjectEvent is the most general. Any event focused on an object derives from this.

IObjectMovedEvent is sent when an object changes location or is renamed. It is quite general, as it also encompasses the case where there's no old location (addition) or no new location (removal).

IObjectAddedEvent and IObjectRemovedEvent both derive from IObjectMovedEvent.

IObjectCopiedEvent is sent just after an object copy is made, but this doesn't mean the object has been put into its new container yet, so it doesn't have a location.

There are only a few basic use cases about what one wants to do with respect to events (but you might want to read the full story in Five/tests/event.txt).

The first use case is the one where the object has to be aware of its path, like in the CoolDocument example above. That's strictly a Zope 2 concern, as Zope 3 has others ways to deal with this.

In Zope 2 an object has a new path through creation, copy or move (rename is a kind of move). The events sent during these three operations are varied: creation sends IObjectAddedEvent, copy sends IObjectCopiedEvent then IObjectAddedEvent, and move sends IObjectMovedEvent.

So to react to new paths, we have to subscribe to IObjectMovedEvent, but this will also get us any IObjectRemovedEvent, which we'll have to filter out by hand (this is unfortunate, and due to the way the Zope 3 interface hierarchy is organized). So to fix the CoolDocument configuration we have to add:

def movedCoolDocument(ob, event):
    """A Cool Document was moved."""
    if not IObjectRemovedEvent.providedBy(event):
        addedCoolDocument(ob, event)

And replace the subscriber with:

...
  <subscriber
      for="Products.CoolProduct.interfaces.ICoolDocument
           zope.app.container.interfaces.IObjectMovedEvent"
      handler="Products.CoolProduct.CoolDocument.movedCoolDocument"
      />
...

The second use case is when the object has to do some cleanup when it is removed from its parent. This used to be in manage_beforeDelete, now we can do the work in a removedCoolDocument method and just subscribe to IObjectRemovedEvent. But wait, this won't take into account moves... So in the same vein as above, we would have to write:

def movedCoolDocument(ob, event):
    """A Cool Document was moved."""
    if not IObjectRemovedEvent.providedBy(event):
        addedCoolDocument(ob, event)
    if not IObjectAddedEvent.providedBy(event):
        removedCoolDocument(ob, event)

The third use case is when your object has to stay registered with some tool, for instance indexed in a catalog, or as above registered with portal_cool. Here we have to know the old object's path to unregister it, so we have to be called before it is removed. We'll use IObjectWillBe... events, that are sent before the actual operations take place:

from OFS.interfaces import IObjectWillBeAddedEvent
def beforeMoveCoolDocument(ob, event):
    """A Cool Document will be moved."""
    if not IObjectWillBeAddedEvent.providedBy(event):
        getToolByName(ob, 'portal_cool').unregisterCool(ob)

def movedCoolDocument(ob, event):
    """A Cool Document was moved."""
    if not IObjectRemovedEvent.providedBy(event):
        getToolByName(ob, 'portal_cool').registerCool(ob)
    ...

And use an additional subscriber:

...
  <subscriber
      for="Products.CoolProduct.interfaces.ICoolDocument
           OFS.interfaces.IObjectWillBeMovedEvent"
      handler="Products.CoolProduct.CoolDocument.beforeMoveCoolDocument"
      />
...

This has to be done if the tool cannot react by itself to objects being added and removed, which obviously would be better as it's ultimately the tool's responsibility and not the object's.

Note that if having tests like:

if not IObjectWillBeAddedEvent.providedBy(event):
if not IObjectRemovedEvent.providedBy(event):

seems cumbersome (and backwards), it is also possible to check what kind of event you're dealing with using:

if event.oldParent is not None:
if event.newParent is not None:

(However be careful, the oldParent and newParent are the old and new parents of the original object for which the event was sent, not of the one to which the event was redispatched using the multi-subscribers we have registered.)

The IObjectWillBe... events are specific to Zope 2 (and imported from OFS.interfaces). Zope 3 doesn't really need them, as object identity is often enough.

CPS Second Bug Day Wrapup

This bug day was announced on short notice, we we still managed to kill 26 bugs (while 9 new were opened). That's good work!

Because there are still 84 active tickets for CPS 3.4 (some are only details of course), we've decided to release a CPS 3.3.7 in the meantime, probably early next week. And we'll keep doing bug days, I propose to have them every Thursday until CPS 3.4 is finally released.

CPS 3.3.6 released

CPS 3.3.6 has just been released.

You'll find download links in the "Quick Download" box of http://www.cps-project.org.

This release contains mainly bugfixes, and some speed improvements. It also bundles for the first time CPSSharedCalendar (successor to CPSCalendar) and CPSMailAccess (sucessor to CPSWebMail), two components heavily based on Five.

CPS Bug Day Wrapup

The CPS bug day this Wednesday was quite productive. A number of actual bugs were fixed, but more importantly a lot of bug triage and sorting happened, to determine what was urgent (going into CPS 3.3.6), what was necessary for the next stable CPS 3.4 release, and what could be postponed to a later development cycle (CPS 3.5). The Timeline for Wednesday show 65 bugs resolved (fixed or closed). In addition to these, several hundreds have been retargetted or cleaned up.

I'm now confident that CPS 3.3.6 will get released within a week. Before CPS 3.4 we still have a number of important things to do though, so there may be a need for a CPS 3.3.7 which would serve as some kind of prerelease, we'll really need "in the wild" validating of some important features such as the CMFSetup-based installation and import/export tools.

The schedule for the remaining work before CPS 3.4 is really conditionned by the availability of the developers. We still expect CPS 3.4 to be released around mid-october.

Object-Relational mapping in Zope

I've been interested in object-relational mapping in Zope for a while. I see this as the keystone of a proper enterprise-grade application server, and I've been investigating existing tools and what can be done with them. Here's a quick rundown.

ZSQL methods

Using ZSQL methods (a standard feature of Zope), you can have your application talk to an SQL database, but you're not really dealing with object-relational mapping, there are no transparently persistent objects. So let's skip it.

SQLObject / sqlos

SQLObject is a pure python mapper, and it is used in Zope 3 through sqlos. They provides declarative mapping for your classes where you can write for instance:

class Person(SQLOS):
    implements(IPerson)
    _columns = [StringCol("username", length=20, notNull=True),
                StringCol("firstname", length=20, notNull=True),
                StringCol("lastname", length=20, notNull=True)]

Then any instance of a Person will actually be stored in SQL behind your back. A unique id is generated for each Person and also stored in SQL. There are various facilities to provide relations between tables, and map them to lists in the python objects.

Archetypes + SQLStorage

SQLStorage is an Archetypes storage that uses SQL as a backend. You can write an Archetypes schema such as:

schema = BaseSchema + Schema((
    StringField("username", required=1, primary=1, searchable=1,
                storage=PostgreSQLStorage(),
                 index="FieldIndex",
                ),
    StringField("firstname", required=1,
                storage=PostgreSQLStorage(),
                ),
    StringField("lastname", required=1,
                storage=PostgreSQLStorage(),
                ),
    )) + TemplateMixin.schema

class PersonSQL(TemplateMixin, BaseContent):
    archetype_name = "Person SQL"
    schema = schema

SQLStorage relies on the Archetypes UIDs to uniquely identify objects. A Relation field can be used to have relations to other objects. Note that Archetypes objects stored through SQLStorage still have a presence in the ZODB, which means it's not a solution if you totally want to get rid of Data.fs bloat.

As you can see in the way things are declared, both SQLObject and Archetypes require you to specify in your code that you will use SQL for some objects. Things are not "transparent" from the programmer's point of view.

Ape

Ape (Adaptable Persistence Engine) is a framework to do object-relational mapping at a lower level than the above two solutions, because it works at the ZODB Storage level. Ape is used as a ZODB Mount Point inside Zope. In fact, Ape is not tied to SQL but is designed to do configurable and flexible mapping to anything; storing to the filesystem is another option provided by default.

Ape relies on the ZODB framework, and especially its classes for persistence and transaction management, backed by the storage APIs themselves. This means that from the programmer's point of view, there's no difference when using Ape than when using a FileStorage or a ZEO ClientStorage. Everything is transparent.

This is a big advantage for the programmer, but the downside is that the structure of the tables in the SQL database is chosen by the framework. However Ape's default SQL mapper already tries hard to provide data mapping in a natural way; for instance all properties are made available in a natural manner, object titles or containment relationship are also naturally expressed.

Hornet

Hornet is an SQL bridge (alpha software for now) that also works at the ZODB layer. In contrast to Ape, it is much more geared toward existing datasets, or regular SQL access to tables. It requires you to define schemas in code too, but once this is done object access is totally transparent, as in Ape. You have to use code like:

PySchema(fieldsets=(
    FieldSet("employee_type", label="Employee Type", widgets=(
        StringWidget("code", label="Employee Code"),
        StringWidget("description", label="Description"),
        )),
    ))

This is then used by the mount point and the underlying Hornet framework for a given table to decide how to map table lines to objects. (The schema is also used to autogenerate forms.)

I want more

To me Hornet and Ape are promising because I believe they integrate at the right level. Ape is better because it doesn't require explicit schema declaration, and that's very important when you have flexible objects where the users add new fields on document instances (which happens all the time in CPS using FlexibleTypeInformation-based content objects).

I believe a large part of the success of the infamous Rails has been that it makes it trivial to interface to an existing SQL database, provided it's regularly named. And I want to bring Zope to that level, but that's just going to be a side effect of a number of other goals.

First, I want to get rid of the catalog and move it to its proper place, which is in a relational database. The ZCatalog is a hack (a successful one I must admit) that tries to bring into the ZODB world something that has been working much better for ages in the relational world. The ZCatalog indexes use a truly gigantic number of small persistent objects to work. Querying an index trashes the ZODB cache which could be used for better things (to cache the objects the user is really accessing). Also the ZCatalog fails to provide any relational features, like JOIN queries. The sorting is weak (you cannot sort on something that's in a full-text index without adding another index). Finally, the "metadata" is duplicating lots information that can be found elsewhere.

If you store your data in SQL, then you can dump the ZCatalog implementation and delegate indexing and searching to SQL.

Second, I want to store blobs in the filesystem. This can be done at the application level by various products such as CPS DiskFileField, Archetypes ExternalStorage, chrism's Blob product, and others. It can also be done transparently at the ZODB storage level using Ape, and that's a much simpler way to do it.

Finally I want my application-specific data (document metadata, document hierarchy, versioning information, various customer specific record-like objects, user definitions, access rights, etc) to be stored in SQL because most of them really live naturally in SQL tables. And because I want to do aggregates on them, like sums and means.

To achieve this, I plan on using the most flexible (and underused) framework available, which is Ape. I'll write various classifiers, and mappers so that a typical CMF or CPS site can be mapped naturally to SQL. I'll also replace the catalog by an implementation that does SQL queries. This will not be a simple endeavour, but I feel this is the only way to get what I truly need in the end, without sacrifying any flexibility.

If Ape proves to hard to work with (because it imposes its own framework of mapping), I'll go the Hornet way of writing a storage directly, with flexible enough policies for the mapping to SQL or the filesystem (or LDAP for that matter).

It goes without saying that this will apply to Zope 3, because that's part of what I think an ECM system should provide. But there's no reason not to make it work in Zope 2, and that's probably where I'll start.

Update: I was asked how this would fit with the "do things in python" philosophy that Martijn has been rightly pushing. Ape is first a pure python library that provides a new database class for use with ZODB (which itself can be considered independent of Zope). Ape also provides Zope-specific mappers that know how to best deal with a number Zope-specific classes. Whenever possible, I'd like to keep to that organisation (also keep in mind that a good separation will make it reusable both by Zope 2 and Zope 3).

ZODB presentation and tips

Chris McDonough has a nice presentation of ZODB, with some tips about how to make it perform better. One I hadn't read about before is the cacheMinimize in the middle of big transactions; I'll have to check how it performs in some situations.

http://www.plope.com/static/presentations/zodb.pdf

A comparison of indexers

http://www.let.rug.nl/~gosse/Imix/clin04_tiedemann.pdf is a poster comparing several Information Retrieval (IR) engines, among them Lucene and Xapian.

This analysis could be helpful in choosing something nice for the Z3 ECM.

Compiling python 2.3.5 on Mac OS X Tiger

Tiger comes with python 2.3.5 bundled, which is nice, but sometimes you want to recompile it, or you want to add site-packages to it without disturbing the standard OS installation. Here's what I had to do to compile it, and python-ldap with it.

Building python 2.3.5:

./configure
vi pyconfig.h
  #undef _POSIX_C_SOURCE
  #undef _XOPEN_SOURCE
  #define HAVE_BROKEN_POSIX_SEMAPHORES
  #define SETPGRP_HAVE_ARG
make
make test
sudo make install

Updated: Added commenting-out of _POSIX_C_SOURCE, otherwise ESHUTDOWN cannot be imported from errno (which makes asyncore fail). Added SETPGRP_HAVE_ARG.

Building Berkeley DB 4.3.28 (needed by OpenLDAP):

cd build_unix
../dist/configure
make
sudo make install

Building OpenLDAP 2.2.26:

CPPFLAGS=-I/usr/local/BerkeleyDB.4.3/include LDFLAGS=-L/usr/local/BerkeleyDB.4.3/lib ./configure
vi config.status
  # replace s%@LIBS@%%g with:
  s%@LIBS@%-lresolv%g
./config.status
make depend
make
sudo make install

Building python-ldap 2.0.7:

vi setup.cfg
  library_dirs = /usr/local/lib
  include_dirs = /usr/local/include
  # comment libs = sasl2 in libs:
  libs = ldap_r lber ssl crypto
/usr/local/bin/python2.3 setup.py build
sudo /usr/local/bin/python2.3 setup.py install

CMFSetup for CPS

I've started playing with CMFSetup to improve importing and exporting the configuration of a CPS site.

CMFSetup is a framework that describes the series of steps needed to setup a CMF site, for instance "setup the actions tool", "setup the workflows", etc. Each step is implemented by code that looks up the details of the configuration in XML files. The CMF Setup Tool itself allows you to execute steps individually (either in import or export mode), to make snapshots of the existing state of a site, to compare them, to revert to any of them, and of course to save the complete configuration to a file.

Yesterday I wrote some code to do the import/export for the directories configuration. I'll do the same soon for schemas and layouts, and then for the rest of the CPS tools. The goal is to be able to instanciate a CPS site entirely through this mechanism, which means that all the configuration would be in XML files.

Amon the remaining questions is, what to do about content, and what to do about things that could be considered configuration but could also be considered content. Instances of the latter are the content of directories (ZODB directories for instance). For pure content stuff, I think a global XML import/export (maybe optionaly retaining only the folder structure but not the documents) can be done.

The advantage of using CMFSetup over CPSIO is that CMFSetup brings a very rich framework for comparison and snapshots. One disadvantage of CMFSetup is that parsing the XML files on import is largely ad-hoc, and painful to code. Using a library like elementtree (like CPSIO does) would improve the process a lot, so I'll play with that.

One nice thing in coding CMFSetup import/export handlers for CPS is that CPSIO has already done this once, so all the code to introspect or recreate objects can be reused, or at least used as an inspiration. Nevertheless it's a lot of work that will have to be done piecewise.

Paris Sprint, Day #2

After a first day of setting up everybody and finding intersting subjects to work on, I spent today working on the Five-2.8 integration.

One of my goals was to remove monkey-patching, which is mainly done except that Five has a need for an unrestricted TALES engine, but Zope's TALES engine is very not modular and very difficult to subclass. Five, to avoid copying code, is actually using code from Dieter Maurer that copies function objects and rebind their arguments to add new definitions to the scope... a nice hack but maybe something we can get rid of.

The goal is to release a beta of Zope 2.8 that includes Five (and the needed Zope X3 components) as soon as possible, possibly by the end of the week (but don't hold your breath).

This will be very useful because it will put Zope 3 technologies directly in the hands of Zope developers, without any additional installs. That's crucial to gain acceptance and visibility, and more importantly to gain developers that will be more willing to work with Zope 3.

Apache + mod_python + ZEO

Pascal Peregrina from lastminute.com is experimenting with using Apache + mod_python as ZEO clients.

Apparently he has good results.

http://www.zope.org/Members/pperegrina/ZopeHandler

http://mail.zope.org/pipermail/zope/2005-February/156379.html
http://mail.zope.org/pipermail/zope/2005-February/156384.html

DeadlockDebugger 1.0

This product adds a hook so that a deadlocked Zope process can be debugged, by dumping a traceback of all running python processes. The dump is sent to the event log (at the DEBUG level) and returned to the browser (even though the Zope is deadlocked and doesn't answer any other requests!).

DeadlockDebugger can of course also be used to debug Zope in non-deadlock situations, when a Zope process is taking a long time and you wish to know what code is being executed.

http://www.zope.org/Members/nuxeo/Products/DeadlockDebugger

Caching is hard

Caching is a difficult problem, especially when needing to correctly invalidate entries. Invalidating is difficult when there may be several caches spread accross ZODB connection (this is the case for '_v_' attributes), or spread accross ZEO clients. Accessing a '_v_' attribute in another ZODB connection (i.e., usually, a different thread) is not really feasible, and accessing another ZEO client that may be accross the world is impossible directly.

Often there are mitigating factors that make the invalidation not really necessary. For instance if the caching is only done for the duration of the current request, an appropriate cleanup of a '_v_' attribute can be done at the end of the request by indirectly using a destructor of the request object. This is what is done by the CMF's MemberDataTool to hold on to the wrapped member objects for the duration of the request, but not longer.

A much better way to do coarse-grained invalidation accross threads and clients is to do it indirectly, through the use of something that can communicate with all these threads and clients: a persistent object. The idea is that all cached values are cached with a generation number, the generation at the moment of their creation. When retrieving a value from the cache, the generation it has is compared to the current generation, and the cached value is deemed invalid if they don't match. This generation number is stored in a persistent object (that is therefore visible to all). When an invalidation must be done, the persistent generation number is incremented (to avoid ConflictError problems, a conflict-resistent class such as BTree.Length should be used). This solution works when a coarse-grained invalidation is suitable, and when don't happen too often.

A bug was recently reported to me that involved caching, but it turned out to be a different class of problem. The problem was that, through a cache, a value holding to persistent references was shared among threads. But it is illegal to use an object from one ZODB connection in another. When threading is involved (which it often is in Zope), problems arise. In this bug, the object itself (a user) was not persisted, but it had to references to persistent objects (the user folder and a directory). In the end I found it better to not store the user object in the cache, but just a dictionnary of information needed to recreate the user object.

Debugging deadlocks

I have been confronted to deadlocks in Zope, where the server was there without taking any CPU but didn't answer any request. It's pretty had to know where to look to find what's happenning.

Maxime Yve found a very useful python module: threadframe. Using it, it's possible to dump tracebacks of all running threads of a python process. I have a specialized dumping product for a client that I'll make generic soon, to debug any Zope.