Lucene-based cataloging solution for Zope-3 and CPS-3.4

I think it's time to drop a note to the outside world about what I've been working on for a little while at Nuxeo. I am pretty confident that this project is nowadays at the end of its first iteration.

This post will give you a short overview of the solution we chose to tackle which is the indexing and searching stack in a Zope and CPS architecture. I submitted an abstract to EuroPython this year. Hopefuly, I'll have the chance to give you more technical details at the conference in July.

Motivations

CPS is based on Zope and the standard cataloging solution of Zope, nowadays, is the ZCatalog. The ZCatalog works really well until a certain amount of indexed documents : that's a fact. As well, ZCatalog extensions, such as TextIndexNG, are of a great interest.

But, because there is a but,  the main problem is that Zope is dealing with a task it shouldn't have to deal with. As a result, it decreases the overall performances of the Zope platform itself. If you are not convinced just try out to inject 200k documents within a Zope instance (or a Plone one if you wish :)) with documents having 50 fields to be indexed and check how your response time is evolving when your instance is as much used by people working and writing within the database and by others consulting it and thus searching all along. In Nuxeo, we tried on large scale projects. It simply doesn't work well/fast enough for serious deployments. Zope gets really slow...

Anyways, you should consider the ZCatalog as what it is : a hack on top of the ZODB because the ZODB doesn't provide any native query language nor full indexing suppport.

For those reasons, we needed such a solution for our customer projects.

As well, this is following our vision of Zope3 being an integration platform for ECM applications where external services could be plugged in thanks to the Zope3 component architecture flexibility and the agility of the Python language.

What is Lucene ?

NXLucene : standalone Lucene indexation server

nuxeo.lucene : Zope 3 cataloging component

nuxeo.lucene is a cataloging component written on top of to the Zope3 application server currently offering an XML-RPC proxy to a NXLucene remote server. As well, It offers an abstraction for Python objects cataloging strategy providing the ability to specify how Python objects should be indexed and retrieved from a Lucene store through NXLucene. (This is important to note here, that whatever remote server providing an XML-RPC remote interface on a Lucene server could be theoretically used.)

Currently, this component is used through Five from CPS. Its integration on top of the Zope3 AS is not finished since we didn't need nuxeo.lucene outside of CPS yet. Feel free to participate to its development if you are interested about having nuxeo.lucene fully integrated on top of a stock Zope3 AS.

nuxeo.lucene is an open source project available under the ZPL part of the CPS platform project.

CPSLuceneCatalog : CMF Catalog replacement for CPS-3. 4

CPSLuceneCatalog is a CPS-3.4.x specific product adding the CPS specific business rules to nuxeo.lucene. For example, it takes care of the way different versions of CPS documents should be indexed. CPSLuceneCatalog is a complete substitute for the ZCatalog that is showing its limits while dealing with millions of objects. CPSLuceneCatalog will be shipped along with the next major release of CPS, version 4, along with the JackRabbit JCR repository.

CPSLuceneCatalog is almost fully backward compatible with the ZCatalog query syntax so be sure you code won't break if you want to migrate. I don't currently support 100% compatibility but I do support at least the subset of ZCatalog query syntax we have been using in CPS internals.

An upgrade step is already available on CPS 3.4.x instances.

CPSLuceneCatalog is an open source project available under the GPL part of the CPS platform project.

Already significant results !

The result is a big win on large scale deployments :

• Indexing and searching are much faster and scalable compared to ZCatalog.

• Indexing and searching are much more powerful compared to ZCatalog (Analysis, ranking, etc...)
  

• Zope global performances are increased because Zope no longer deals with the indexing and searching business.
• 
  

Looking for support ?

If you are looking for any technical information or help regarding these products please subscribe to the CPS devel mailing list.

If you are looking for commercial support, Nuxeo provides professional services whatever your needs are.

Nuxeo is currently maintaining NXLucene, nuxeo.lucene and  CPSLuceneCatalog and we are always welcoming third-party contributors. As a developer, if you are interested about contributing to these projects, we will grant you access to our svn repositories and provide you all the information you need in order to get started. Just subscribe to the CPS devel mailing list.

Thanks

A big thanks to our customers at Nuxeo for trusting us, being patient and for always bringing along with, their projects, bleeding edge use cases.

And don't forget, at Nuxeo we love challenge and innovation !

Hope you'll enjoy those components as much as I enjoyed writing them for our customers. Looking forward to hearing from you.

    J.