By now, most of you should have heard about CMIS, the upcoming specification that promises interoperability between many systems for common content management tasks. The CMIS specification is being driven by an OASIS Technical Committee and is currently still a draft; it is expected to be finalized late 2009 or early 2010.

I won't detail here all that CMIS will bring, this has been covered extensively already and will be even more in the future... No, the purpose of this article is to present Chemistry.

Chemistry

Chemistry is a new Apache project for CMIS that started incubating recently ("incubation" is the term used in the Apache Software Foundation for young projects that still have to prove themselves). Chemistry's goal is to provide general purposes libraries for interaction using CMIS between a server and a client. These libraries are mainly written in Java, but some JavaScript code has been added as well, and we're open to more.

Chemistry provides a high level API so that a developer can manipulate objects like documents or folders and can call simple methods on them without having to deal with details of a specific low-level communication transport. In addition to that, Chemistry also provides a SPI (Service Provider Interface) for backend developers, making it quite easy to use Chemistry to store documents in a project-specific manner.

Underlying this, Chemistry has implementations for the CMIS transports. CMIS specifies two mandatory transport protocol bindings (one extending AtomPub, for a lightweight RESTful HTTP interface, and another using SOAP for a WebService-based interface), and Chemistry will support both — and probably more in the future.

The current Chemistry code base has an initial version of the API/SPI together with some actual implementations around the AtomPub protocol. Already Chemistry can talk to itself (AtomPub client talking to AtomPub server) and store data in-memory (which is very handy for unit tests). Outside of the Apache code base, Nuxeo has also coded a backend to provide access to Nuxeo 5.2 repositories using Chemistry. Generic CMIS AtomPub clients like CMIS Explorer are able to see a Nuxeo repository through Chemistry for instance.

Chemistry Modules

The following modules will be available in Chemistry:

• The APIs: a low-level SPI between a client and a server that mirrors the CMIS specification closely (it is expected that the SPI will be used when either the client or the server implements one of the HTTP protocols defined in CMIS), and a high-level API that wraps the SPI to provide more object-oriented notions of connections, folders and documents, and that hides the nitty-gritty details of the protocols.
• A set of common Java utilities around CMIS, for instance a parser to turn CMIS SQL into an AST (Abstract Syntax Tree) that can be reused by different backends, or a generic in-memory implementation of the SPI and API for unit testing.
• Four implementations of the SPI for the protocols defined by CMIS: an AtomPub server and client, and a SOAP server and client.
• A generic implementation of the API-to-SPI wrapping, so that a third-party implementation of just the SPI can be plugged into the rest of the Chemistry framework. (Some of the four basic protocol implementations may also provide the full API when this is more efficient than using the generic wrapping.)
• An implementation of the APIs as a JCR backend.
• A set of generic tests for CMIS servers and client, providing an unofficial TCK for CMIS.

In the future, it is expected that more implementations of the APIs will be available, for example we envision new transports:

• A WebDAV-based transport.
• An HTTP-based transport less RESTish and more friendly to browsers and JavaScript.

And new backends:

• A backend storing documents on the filesystem, with or without metadata.
• A backend storing documents in the Google AppEngine Datastore.
• A backend storing documents using Microsoft Windows SharePoint Services.

The Pieces of the Puzzle

As you can see, these modules will allow for wide interoperability between systems. Here's a graphical representation of the building blocks:

The User Application speaks the API:

The API can be implemented in many ways. First, it could be a direct backend:

Or, more commonly, the API will be implemented as a client binding for a specific protocol, SOAP of AtomPub:

Each protocol speaks in its own way on the wire:

And this is connected to a server that speaks the protocol as well:

Finally, behind the server, a backend has to store the actual information somewhere:

Anyone is welcome to create new pieces, for instance new protocol bindings:

Or new storage backends:

Now let's see how the main pieces can be plugged together.

The simplest connection is between an application and a direct backend:

If the backend only wants to deal with the SPI, its implementation can reuse the API-to-SPI to provide a full API experience:

When talking through a wire protocol, we plug together a client and a server:

The end result is an application talking to a backend through a wire protocol:

Of course we can get creative and plug many more together:

Development

All of this is still a work in progress (even the spec!), but you should expect rapid changes in the available features in the coming months as the spec settles down, more code is written, more test cases are written, and more testing against third-party implementations is done.

If you're interested in helping, please join the list chemistry-dev@incubator.apache.org by sending an empty email to chemistry-dev-subscribe@incubator.apache.org.

A few weeks ago I gave an interview to Irina Guseva of CMSWire. We touched the subjects of strategic value of CMIS, Apache Chemistry project history, partnerships, open source, future plans around CMIS, and more.

Chemistry has extremely ambitious plans. We believe that it can become the de facto bridge between most of the Java-based content-oriented products, allowing a very wide variety of back-ends and applications to be connected together. And actually Java is not the sole language that this project is targeting, as David Nuescheler is also working on a JavaScript library for CMIS. In the coming month you should see an exponential increase in the functionality that Chemistry provides...

You can read the full article at CMSWire.

Schedule

Existing capabilities

Retention & Hold

Tagging

• Adding metadata to a tag: tag date and author are typical of the use case here. The tags are then seen as either a "rich" property, or as a relationship (carrying metadata) to a concept in a taxonomy.
• Tagging an object on which one doesn't have write access: this is quite common and typical of the bookmarking community, think "del.icio.us". Of course to allow this you can't use a basic MVP as you don't have write access to the document being tagged.
• Does tagging an object change its last-modification date? This may be constrained by the implementation of the tags.
• Changing many tags at the same time. This is typical of the "tag normalization" use case, where a TagMaster has determined that two tags have to be merged. Some batch modification features may be useful in this case.
• Querying tags for things like: most common tags, less used tags, tags most used together. Also querying for documents in relation to their tags, for instance documents sorted by total tag weight, documents with the most tags, documents most recently tagged. Or even people having added most tags, etc.
• Maintaining the tag cloud, or taxonomy: in a system with many tags, maintaining the tag cloud becomes a full-time job. Having relationships between similar tags, merging them, weighting them, is important.

Transactions

Events and notifications

REST

RI/TCK

ACLs

Search

Next steps

• What is Enterprise Content Management
• What do we want to achieve
• Core Framework
• Core and High-Level Services
• UI Layers
• Extensible and Pluggable
• Using the Framework

Nuxeo is switching its ECM to Java, and we're using JCR for our document storage. JCR (Java Content Repository, standardized by JSR-170 and the upcoming JSR-283) is a young specification with a promising future — but what's its point, you may ask, as all existing content management systems are already storing content very well without it? Its goal is interoperability between vendors, which will make it possible for people who write applications needing to store content to have a unified API for such manipulations. All major content repository vendors are active in the JSR-283 expert group, and all are working on JCR bindings for their various proprietary repositories.

Of course a standardized and wildly successful way of manipulating content already existed before JCR: SQL. But SQL and JCR have a different focus:

• SQL is a language; it manipulates rows and is geared toward generic relation manipulation,
• JCR is a Java API; it manipulates nodes and is geared toward hierarchical manipulation (parent-children).

SQL and JCR have quite different underlying data models:

• SQL's model is that of tables with fixed schemas, and relations between tables,
• JCR's model is that of a tree of nodes with flexible schemas and with parent-children relations as the main focus — although other types of relations exist.

JCR also offers higher level features than SQL, notably workspace and version management.

For many kinds of applications, there is a focus on being able to arrange documents in folder hierarchies, and to have a wild variety of structure for these documents. In this case, a storage model based on JCR is much more suited than something based on SQL.

It should be noted that many things in the computing world already are based on the notion of folder hierarchies storing arbitrary documents:

• filesystems are a tree of (unstructured) documents,
• Revision control systems are based on filesystem concepts but add a lot of structure on top of them,
• WebDAV (along with DAV and DELTAV) is a protocol that addresses documents using a path, and where documents have a flexible set of properties,
• most proprietary content management systems are based on (or offer the notion of) organizing the content in a hierarchy,
• Nuxeo's CPS is itself based on this classic model.

This is why JCR has emerged as a common and useful storage API for all these use cases.

For an ECM framework like Nuxeo 5, JCR interoperability can be seen from two different directions:

• JCR provides an API that we can use to store our content, to make us vendor-independent and flexible regarding storage,
• JCR provides an API through which we can expose our content, which makes our platform usable by any external system that understands it — we're ourselves a vendor providing JCR bindings.

For its initial release, Nuxeo 5 is focusing on the use of JCR as its main storage implementation and uses Jackrabbit to store most of our content. In the future, we'll also provide JCR bindings so that the high-level content we provide can be directly accessed from external applications using JCR too.

In a content management system, the actual data that the system or the users manipulate comes from many kinds of sources. Content can come from a JCR repository, or from a relational database, or from an LDAP directory, or from a semantic storage engine like Jena, or from any other kind of open or proprietary storage engine.

But fundamentally all these kinds of content, which I'll call "records", aren't very different:

• a record can be created, viewed, modified, deleted,
• a record can often be copied or moved,
• a record obeys a schema that can be known to the system, this means that its individual fields are strictly typed,
• when being viewed or modified, a record has a user interface that is based on forms, labels, widgets, depending on the schema,
• records can be searched and a result set returned,
• records can be listed in a compact form (search results, folder contents, user dashboard, workflow workitems, RDB table listing, user information browsing, etc.),
• records have an identity (like a unique id) or a location (like a path), sometimes both.

One of the strengths of CPS is to use a common abstraction for many of these concepts, embodied in the CPSSchemas component. In Nuxeo 5 we want to go further and provide even more integration for all these, the base components for these abstractions are NXCore and NXTypeManager.

The reasons to strive for convergence are numerous:

1. this merges into unique concepts things that had been previously separated because of different implementation choices. For instance, an LDAP schema is not fundamentally different from an SQL schema (or from an XML Schema if one is interested in the relevant subset).
2. this gives the programmers a common API for all data-related operations, which means more reusability. For instance changing an attribute in an LDAP entry doesn't have to be different from changing the title of a document or changing a value in an RDB row; processing a list of search results to display them in a table doesn't have to be different from processing the children of a folder to display the folder's contents.
3. this gives the framework developers a way to optimize some operations because of commonalities in the underlying implementations. For instance you don't need three kinds of events dispatching for "LDAP entry modified", "RDB row modified" or "document modified".
4. this gives the users a unified way of manipulating different kinds of data when there's really no need to have a different UI for them. When a user fills a form it's really the same process whether he's modifying his personal preferences, adding a keyword to a document, or changing a quantity in an RDB row.
5. this allows very simple migrations between storage technologies, when these are felt necessary. A customer could start with an LDAP database for its user base and later have the need to move them to an RDB table. User entries in an RDB table may need to be versioned and moved to a JCR storage. An application should survive all these with only configuration changes and no code to rewrite.

It should be noted that this means that JCR is in no way the primary storage model for Nuxeo 5, it's only the first one to be implemented. In the future, it will be possible to store documents in LDAP or an RDB. When a suitable storage model is devised and implemented, you'll be able to apply workflow or versioning to RDB-based documents for instance.

This convergence is quite exciting to us, and our goal is to allow people to build complex applications with Nuxeo 5 in a more straightforward manner.

GenericSetup is a framework to describe the configuration of a Zope site as a set of XML files (and sometimes other associated files). It can import profiles, which may create objects or change their configuration, and export profiles, which makes a snapshot of the configuration and writes it to a set of XML files.

GenericSetup provides a tool that can store snapshots of a configuration in the ZODB itself, where it can be examined and even modified. It can also do diffs between two snapshots, which is very useful to find out what changed in a configuration (it's a good idea to take a full snapshot anytime some significant changes are made to the configuration).

GenericSetup differentiates between Base and Extension profiles. A Base profile is a profile that describes "everything". When it is imported, it removes and overwrites any previous configuration. An Extension profile is a profile designed to be incrementally added on top of a previously existing configuration. It may of course overwrite some settings, or even in some case remove objects, but its goal is generally to add optional configuration on top of a main one

GenericSetup is based on a small number of concepts: the toolset, and some import and export steps. GenericSetup provides a framework where import and export steps can be written simply using Zope 3 adapters.

GenericSetup was born as "CMFSetup" but was later made generic and can be used by any Zope 2 application. It is now available at http://svn.zope.org/GenericSetup/trunk/. A subclass with additional features is used in CPS.

<?xml version="1.0"?>
<tool-setup>
  ...
  <required tool_id="portal_trees"
            class="Products.CPSCore.TreeTool.TreeTool"/>
  ...
</tool-setup>

<?xml version="1.0"?>
<import-steps>
  ...
  <import-step id="trees" version="20051230-01"
               handler="Products.CPSCore.exportimport.importTreeTool"
               title="Tree Tool">
    <dependency step="toolset"/>
    Import tree tool and tree caches.
  </import-step>
  ...
</import-steps>

<?xml version="1.0"?>
<export-steps>
  ...
  <export-step id="trees"
               handler="Products.CPSCore.exportimport.exportTreeTool"
               title="Tree Tool">
    Export tree tool and tree caches.
  </export-step>
  ...
</export-steps>

def exportTreeTool(context):
    """Export Tree tool and tree caches as a set of XML files.
    """
    site = context.getSite()
    tool = getToolByName(site, 'portal_trees', None)
    if tool is None:
        logger = context.getLogger('trees')
        logger.info("Nothing to export.")
        return
    exportObjects(tool, '', context)

def importTreeTool(context):
    """Import Tree tool and tree caches as a set of XML files.
    """
    site = context.getSite()
    tool = getToolByName(site, 'portal_trees')
    importObjects(tool, '', context)

<adapter
    factory=".exportimport.TreeToolXMLAdapter"
    provides="Products.GenericSetup.interfaces.IBody"
    for=".interfaces.ITreeTool
         Products.GenericSetup.interfaces.ISetupEnviron"
    />

from zope.interface import Interface

class ITreeTool(Interface):
    """Tree Tool.
    """

from zope.interface import implements

class TreeTool(UniqueObject, Folder):
    """Tree Tool that caches information about the site's hierarchies.
    """
    implements(ITreeTool)

    id = 'portal_trees'
    meta_type = 'CPS Tree Tool'
    ...

<cps:upgradeStep
    title="Replace CMF URL tool with a CPS-specific one"
    source="3.3.4" destination="3.3.5"
    handler=".upgrade.upgradeURLTool"
    checker=".upgrade.check_upgradeURLTool"
    />