Julien Anguenot

ICSSEA 2006 : Nuxeo 5 talk this thursday

janguenot — 2006-12-05

I will be giving a talk at the 19^th International Conference "Software & Systems Engineering and their Applications", about ECM and Nuxeo 5, this thursday in Paris.

You can find the detailed program of the conference here. The conference will be held by the CNAM. You can find information about the location over there.

See you there

Java soon to be opened !

janguenot — 2006-10-28

Here is a good news for Nuxeo 5 : Sun CEO sets open source Java time frame

As we read in a few discussion threads after the release of NXRuntime and Nuxeo Core, some people were pointing out that, with our Java switch, we weren't using a full open source stack anymore. These discussions will be over soon.

As well, some people asked us if we considered making Nuxeo 5 running using gcj. Well, ... we have been trying to leverage gcj with the NXLucene project where we were using Lucene through gcj. This is working ok but what a pain... It appeared to be extremely complicated to find the right version of gcj depending on the target platform on production and that caused us lots of worries... So definitely not a viable option. Ask the hosting companies if you are not convinced ... As well, a lot of work is required at application level while trying to port gcj-based application on upper versions of gcj because of bugs in the compiler itself. So much frustration...

Furthermore, we can expect Java enabled Linux distributions out of the box in the future with Java applications pre-installed and shipped with the distribution itself : for instance JBoss AS.

Because of the fact that Nuxeo bas been caring about open source for more than 6 years, playing the open source game and choosing a core business fully based on a transparent open source model, we are more than happy that this time frame set announcement comes up almost at the same time as the release of Nuxeo 5.

Nuxeo Core 1.0 released !

janguenot — 2006-10-25

I'm proud to let you know that we released Nuxeo Core 1.0 ! This is the first release of the new embeddable document management core that is the foundation of the upcoming Nuxeo 5 ECM platform.

Nuxeo Core is an embeddable document management core, based on Nuxeo Runtime. It provides all necessary low-level services to define, store, manage, audit, request and search content. It is the kernel of Nuxeo 5 and can also be embedded in third-party applications to provide advanced content management features. It can run on any Java platform, can be easily extended using plug-ins (thanks to Nuxeo Runtime) and implements JCA (Java EE Connector Architecture) to be easily plugged into existing applications or information systems.

Nuxeo core key features :

Model: content management concepts definition
Storage Manager: low-level content storage and retrieval (JCR-based)
Schema Manager: XMLSchema based content schema definition and management
Version Manager: versioning management
Content Type Manager: content types management (a content type is a set of schemas)
Query Engine: SQL-like unified query engine
Security Manager: access rights manager and controller
Life cycle management: define, deploy and applies life cycle on document types (JCR storage)
Event Service: abstract event service to broadcast and receive events
Local (POJO) and Remote (JavaEE) access using dedicated client sessions.
Set of extension points allowing third party code to extend and customize the Nuxeo Core services

You will find detailed information here. This is a detailed technical overview of Nuxeo Core 1.0.

A JBoss binary release is available along with a simple Eclipse RCP demo application that leverages the Nuxeo Core. (The demo application is available for Linux, OSX and Win32). You will find in this document all the needed information related the Eclipse demo application.

Check the project homepage for more information and especially if you want to check the source code.

This is an important step toward the release of Nuxeo 5 planned in the middle of next month. We are all more than happy and proud of this new platform and we hope you'll enjoy it as well !

Stay tuned there is a lot in the pipe ! :)

Switching from Emacs to PyDev...

janguenot — 2006-09-15

I gotta admit I was wrong...

I recently tried out Pydev and Pydev extensions on Eclipse 3.2 for my
Python and Zope developments. Wouah !

I've been using these plugins for couple of weeks and I already can't
switch to emacs anymore. I wouldn't have believed It could have ever
happened. I've been using emacs for years and somehow refused to try
out any IDE for Python because I thought it wasn't as crucial for
Python development as for other languages such as Java, C#, C++
etc....

I love this feeling. I mean when you find out a tool and are wondering
how you've been living without it for so long that you can't go back to
the old one :) I'm in this case with emacs. I simply can't
anymore. (Well, I could but you see what I mean)

Pydev really brings a lot. I was surprised by the progresses made by
Fabio on the Pydev front and especially what Pydev extensions bring
related to code completion.

I let you check out the screencasts to get a quick overview if you
don't have time to install and play with it (or you are not convinced)

For Zope development there are some things I'm missing that require
some command lines beside Pydev. (Or these are maybe some things I
didn't find since I'm relatively new in Pydev development)

Support for zope.interface

      This is a bit annoying since pydev doesn't recognize
      zope.interface and thus shows errors on those classes. I might
      even dream about a full integration checking the Python classes
      implementing interfaces such as in Java. And yeah, I'm still
      dreaming about builtin Python interfaces but eh wait ... Python
      guys don't want that...I need to stop dreaming...

Support for doctests

      Pydev doesn't support doctests. Although, you can run external
      commands from Eclipse. I'm using them to specify a zope.testing
      based tests.py as external command and I get back the result in
      the eclipse console. It would be a plus if it could directly run
      them from the Eclipse as doctests as the other Pyunit tests.

Python specific package browser

      Here, I'm especially talking about the refactoring. Move and
      rename you can find in the Java package manager. So that you can
      rename module names and don't worry about changing the references
      in other Python projects in the workspace. Or still move modules
      with the same behavior. It really does ease the refactoring in
      Java.

Zope server plugin
Debugger integrated with Zope

To sump up, Pydev and Pydev extensions rock. It's really promising and
it addresses a huge lack related to Python development.

Please don't tell me about other Python specific IDEs since this is
not what integrators are expecting. They already do have Eclipse on
theirs boxes all over the place so they just want a new Eclipse
plugin. (and they are right) Another industry reality. I guess IDE
war is over ... as well.

I guess �ric is going to like this post ;)

EuroPython 2006 slides : A Lucene-based cataloging solution for Zope-3 and CPS-3.4

janguenot — 2006-07-05

I am just back from EuroPython where I did a presentation about NXLucene and the Zope / CPS Platform integration, yesterday the 4th. You can grab the slides here.

EuroPython 2006 : timetable of the conference partially announced

janguenot — 2006-06-06

The timetable of EuroPython 2006 has been partially published yesterday night. You can check it out over t here.

Nuxeo will be there :

I will be talking about Lucene-based cataloging solution for Zope-3 and CPS-3.4. on thuesday the 4th. See my recent post related to this. I am really excited to present these pieces of software to the Python / Zope community !
Tarek will be talking about CPSBayes : a multipurpose naive bayesian classifier for Zope the same day in the morning.

Because of a really busy schedule nowadays at Nuxeo, I won't be able to attend the whole conference and will be in Switzerland on thuesday only. I will certainly attend the conference dinner in the evening. I will be pleased to meet anyone interested about discussing anything related to Python, Zope or Nuxeo.

See you there !

St�fane just switched to Mac !

janguenot — 2006-06-05

Huge happening today at Nuxeo : Stefane just switched to Mac !

Here is one picture taken in the wild with my phone ;)

Lucene-based cataloging solution for Zope-3 and CPS-3.4

janguenot — 2006-06-04

I think it's time to drop a note to the outside world about what I've been working on for a little while at Nuxeo. I am pretty confident that this project is nowadays at the end of its first iteration.

This post will give you a short overview of the solution we chose to tackle which is the indexing and searching stack in a Zope and CPS architecture. I submitted an abstract to EuroPython this year. Hopefuly, I'll have the chance to give you more technical details at the conference in July.

Motivations

CPS is based on Zope and the standard cataloging solution of Zope, nowadays, is the ZCatalog. The ZCatalog works really well until a certain amount of indexed documents : that's a fact. As well, ZCatalog extensions, such as TextIndexNG, are of a great interest.

But, because there is a but, the main problem is that Zope is dealing with a task it shouldn't have to deal with. As a result, it decreases the overall performances of the Zope platform itself. If you are not convinced just try out to inject 200k documents within a Zope instance (or a Plone one if you wish :)) with documents having 50 fields to be indexed and check how your response time is evolving when your instance is as much used by people working and writing within the database and by others consulting it and thus searching all along. In Nuxeo, we tried on large scale projects. It simply doesn't work well/fast enough for serious deployments. Zope gets really slow...

Anyways, you should consider the ZCatalog as what it is : a hack on top of the ZODB because the ZODB doesn't provide any native query language nor full indexing suppport.

For those reasons, we needed such a solution for our customer projects.

As well, this is following our vision of Zope3 being an integration platform for ECM applications where external services could be plugged in thanks to the Zope3 component architecture flexibility and the agility of the Python language.

What is Lucene ?

Lucene is an open source project from the Apache Software Foundation written in Java. This is a high-performance, full-featured text search engine library.

I would suggest that you check the website that contains a lot of useful information and documentation. As well, I would really recommand this book to anyone interested in working with Lucene and / or in understanding more deeply how it works and how to use it in a proper way. As well, some projects such as nutch are described as case studies which is more than interesting for anyone who wants to build a system on top of Lucene since the best practices are described within those case studies.

In Nuxeo, we first integrated Lucene for a customer within the Apogee project scope. (Apogee is a framework based on Eclipse RCP for ECM rich client applications). Its use had been a real success so we decided to go further and see how we could leverage the use of Lucene server side.

What is PyLucene ?

The first time we've seriously considered using PyLucene was at last year's EuroPython conference after Andi Vajda's really great presentation of PyLucene. Andi is the actual main PyLucene developer. PyLucene is maintained by the Open Source Applications Foundation.

PyLucene is a GCJ-compiled version of Java Lucene integrated with Python. Its goal is to allow the use of Lucene's text indexing and searching capabilities from Python. It is designed to be API compatible with the latest version of Java Lucene.

PyLucene is freaking fast ! Even faster than the Java Lucene version according to the authors of the Lucene In Action book. Furthermore, It will be easily synchronized with the latest Java Lucene releases since this is not a from scratch port but a GCJ-compiled version of Java Lucene itself.

NXLucene : standalone Lucene indexation server

NXLucene is a standalone multi-threaded remote server handling Lucene stores. It takes advantage of the freaking fast PyLucene Python bindings and uses Twisted for its server implementation. It uses some part of the Zope3 component architecture as well. NXLucene currently supports the XML-RPC protocol. (Its roadmap includes an ICE connector for the 1.x branch.) As well, NXLucene might be seen as a good example of what could be achieved using the best parts of different worlds (Java Lucene , PyLucene, Zope3, Twisted,...). Bear in mind, that NXLucene is not running on top of the Zope AS. It is standlone.

NXLucene exposes an XML query language for indexing and searching operations. Note the Lucene native search query is of course still supported. Check the NXLucene interfaces

While installing NXLucene, you will install as well the core libs that might be used by third party Python programs. For instance, the query lib might be useful to help you format your NXLucene XML queries or still the testing library might be really helpful to write tests for your Python components that need to communicate with an NXLucene server.

This is important to note here that you can request NXLucene using any language. You will only need an XML-RPC client library to do so.

NXLucene is an open source project under the LGPL part of the CPS platform project.

For more information about NXLucene and its installation you may check the NXLucene website.

nuxeo.lucene : Zope 3 cataloging component

nuxeo.lucene is a cataloging component written on top of to the Zope3 application server currently offering an XML-RPC proxy to a NXLucene remote server. As well, It offers an abstraction for Python objects cataloging strategy providing the ability to specify how Python objects should be indexed and retrieved from a Lucene store through NXLucene. (This is important to note here, that whatever remote server providing an XML-RPC remote interface on a Lucene server could be theoretically used.)

Currently, this component is used through Five from CPS. Its integration on top of the Zope3 AS is not finished since we didn't need nuxeo.lucene outside of CPS yet. Feel free to participate to its development if you are interested about having nuxeo.lucene fully integrated on top of a stock Zope3 AS.

nuxeo.lucene is an open source project available under the ZPL part of the CPS platform project.

CPSLuceneCatalog : CMF Catalog replacement for CPS-3. 4

CPSLuceneCatalog is a CPS-3.4.x specific product adding the CPS specific business rules to nuxeo.lucene. For example, it takes care of the way different versions of CPS documents should be indexed. CPSLuceneCatalog is a complete substitute for the ZCatalog that is showing its limits while dealing with millions of objects. CPSLuceneCatalog will be shipped along with the next major release of CPS, version 4, along with the JackRabbit JCR repository.

CPSLuceneCatalog is almost fully backward compatible with the ZCatalog query syntax so be sure you code won't break if you want to migrate. I don't currently support 100% compatibility but I do support at least the subset of ZCatalog query syntax we have been using in CPS internals.

An upgrade step is already available on CPS 3.4.x instances.

CPSLuceneCatalog is an open source project available under the GPL part of the CPS platform project.

Already significant results !

The result is a big win on large scale deployments :

Indexing and searching are much faster and scalable compared to ZCatalog.

Indexing and searching are much more powerful compared to ZCatalog (Analysis, ranking, etc...)

Zope global performances are increased because Zope no longer deals with the indexing and searching business.

Looking for support ?

If you are looking for any technical information or help regarding these products please subscribe to the CPS devel mailing list .

If you are looking for commercial support, Nuxeo provides professional services whatever your needs are.

Nuxeo is currently maintaining NXLucene, nuxeo.lucene and CPSLuceneCatalog and we are always welcoming third-party contributors. As a developer, if you are interested about contributing to these projects, we will grant you access to our svn repositories and provide you all the information you need in order to get started. Just subscribe to the CPS devel mailing list.

Thanks

A big thanks to our customers at Nuxeo for trusting us, being patient and for always bringing along with, their projects, bleeding edge use cases.

And don't forget, at Nuxeo we love challenge and innovation !

Hope you'll enjoy those components as much as I enjoyed writing them for our customers. Looking forward to hearing from you.

J.

Slides of my Zope3 talk at the www2006 in Edinburgh

janguenot — 2006-05-31

You can find here the slides of the Zope 3 talk I gave last friday at the www2006.

The talk went well. (The time they gave me was a little shorter than I would have expected though). This was the last day of the conference thus, as always during conferences, lots of people had already left the place.

I was surprised to find a room with lots of people knowing Zope2 (or at least people who had tried Zope2 couple of years ago). The bad news was that they seemed not to like it... (Actually, like a lot of people outside of the Zope world). Arguments are always the same, too monolitic, too complex , not enough following standards etc...

I guess I succeeded to excite their curiosity with Zope3 and the way it's been designed to especially tackle the issues they had previously with Zope2. They really loved the doctests :)

The www2006 conference is about the Web in general right :). So basically, I found there lots of people excited about Web 2.0, AJAX and all browser related technologies. Of course, I can see the advantages of those technologies but I have to say I don't think I'm a Web guy at heart. I would always rather work server side or with real desktop applications myself :)

Edinburgh is a nice big village. People were really nice (and they do have a really cool English accent ;)). The only problem with small cities is for instance when you would like to eat after 10 P.M... Kudos to the Indian restaurant next to the hotel for serving me food when I arrived from the airport even if they were about to close when I came in...

I didn't spend a lot of time in Edinburgh since I went to London on Friday evening.

Here it is for this small report about the conference.

Dump and load foreign subversion repositories using svnadmin and svndumpfilter

janguenot — 2006-04-21

What for ?

This is mostly a reminder for mysef about the way of dumping and loading subversion repositories.

Let's say I have two subversion repositories. repoA and repoB. I got a component called P at the root of repoA as below :

repoA /

          P /

             trunk /

             branches /

             tags /

What I'd like todo is to copy P from repoA to repoB preserving all the history logs for P.

Dumping the repository

Let's say I started the developement of P with revision 28000 and my last checkin related to P is at revision 38000. The revision numbers are important because it will decrease the needed time to dump the actual svn repository. Less revisions to dump means less time needed :)

We will use svnadmin to do that.

Go on the server hosting the subversion repository and dump it this way :

$ svnadmin dump --quiet --revision 28000:38000 /home/svn/repoA > repoA.dump

This operation can last a while depending on the size of your repository.

svnadmin dump options :

$ svnadmin dump --help

dump: usage: svnadmin dump REPOS_PATH [-r LOWER[:UPPER] ] [--incremental]



Dump the contents of filesystem to stdout in a 'dumpfile'

portable format, sending feedback to stderr.  Dump revisions

LOWER rev through UPPER rev.  If no revisions are given, dump all

revision trees.  If only LOWER is given, dump that one revision tree.

If --incremental is passed, then the first revision dumped will be

a diff against the previous revision, instead of the usual fulltext.



Valid options:

  -r [--revision] arg      : specify revision number ARG (or X:Y range)

  --incremental            : dump incrementally

  --deltas                 : use deltas in dump output

  -q [--quiet]             : no progress (only errors) to stderr

Filtering P component from repoA's dump

What we are interested in here is only component P. We need to filter out for P component. We will use svndumpfilter to do that.

$ cat repoA.dump | svndumpfilter include P > P.dump

svndumpfilter options :

$ svndumpfilter --help

general usage: svndumpfilter SUBCOMMAND [ARGS & OPTIONS ...]

Type 'svndumpfilter help <subcommand>' for help on a specific subcommand.



Available subcommands:

   exclude

   include

   help (?, h)

Loading back P dump to repoB

Now go on the server hosting repoB. Before that, transfer your P.dump to this server. We will use svnadmin to load it back

$ svnadmin load --quiet /home/svn/repoB < P.dump

svnadmin load options :

$ svnadmin load --help

load: usage: svnadmin load REPOS_PATH

Read a 'dumpfile'-formatted stream from stdin, committing

new revisions into the repository's filesystem.  If the repository

was previously empty, its UUID will, by default, be changed to the

one specified in the stream.  Progress feedback is sent to stdout.

Valid options:

  -q [--quiet]             : no progress (only errors) to stderr

  --ignore-uuid            : ignore any repos UUID found in the stream

  --force-uuid             : set repos UUID to that found in stream, if any

  --use-pre-commit-hook    : call pre-commit hook before committing revisions

  --use-post-commit-hook   : call post-commit hook after committing revisions

  --parent-dir arg         : load at specified directory in repository

Here we are. You just need to remove P from repoA

$ svn remove http://svn.foo.com/repoA/P

Of course, what would be really cool, instead of the dump/load
operations, would be something like that being possible in between foreign repositories

$ svn move https://svn.foo.com/repoA/P https://svn.foo.com/repoB/

Thanks Florent for the pointers.