Nuxeo Developers Blog

Zope community portraits, meet the guys

I have started to collect portraits from Zope community people on this page. Meet them !

(Post originally written by Tarek Ziadé on the old Nuxeo blogs.)

Diving into Funambol

In order to understand Sync4J (also named Funambol), there are many hooks to use. First of all, the funambol data service server developer book to understand the architecture. After reading this paper, I needed to completely dive into Sync4J code. Hopefully, they provide an email connector to have an example.

From now and in the following Sync4J posts, I'll write some notes about Sync4J development and difficulties I've met.

• First problem was to find how to get the credentials specified in the SyncML specs to authenticate on the JCR. Here is an interesting piece of code inspired from the email connector source:

String username = ((Sync4jPrincipal)principal).getUsername();
String credentials = ((Sync4jPrincipal)principal).getEncodedCredentials();
String password = Helper.getPassword(username, credentials);

The credentials are formatted in the following way: "username:password" and encrypted. To decrypt, there is only to use the Base64.decode() method. The login to the repository will be done during the synchronization initialisation (beginSync())

• An other point to think about is to create a testing framework in order to test the connection between Sync4J and a JackRabbit implementation of JCR.
  

(Post originally written by Cedric Bosdonnat on the old Nuxeo blogs.)

Open Source ECM: CPS Platform 4 "YellowCake" Teaser

CPS 4 Architecture Overview

CPS Platform v3.4 is already the most complete Enterprise Content Management platform in the open source market. Its large feature scope offers a great choice of functional components to address the whole ECM scope. This leads CPS to be used for many different document-centric applications like paper and electronic mail management for large organizations, pure document management for the nuclear industry, collaboration platform, civil-state acts management, web content management, knowledge base management, record management, intranets, and a lot more...

Our R&D team has been working for a few months on the next major release of CPS: CPS 4 (codenamed YellowCake). The goals are to open CPS to other technologies, leverage major industry standards, enable high-end scalability and keep all the good features and technical coherence already present in the framework. We have worked a lot on prototyping and experimenting some deep improvements to the core infrastructure.

I am really excited to present and open it to the community now, after some time of raw-thinking and prototyping. I hope you will like what you will read below as much as I do. Do not hesitate to send comments on the mailing-list.

Goals

Here are YellowCake's main targets :

• High-end scalability: 10+ million documents, 5+ terabytes of data, thousands of users in one instance.
• Open, Integrate and Extend: enable developers to write CPS components in all major languages and technologies and allow the use of an external message broker (JMS) to plug CPS in the enterprise's information system. Creating components in plain old Python (POP :-), Java, C# or PHP that can react to events and/or control CPS will be dramatically easy. Let customer and developer choose their technologies!
• Definitely standard-based: XMLSchemas to describe content types, JCR level2 compliant repository, JSR-168 portlets for integration in J2EE portals, XForms for content forms, web-services, RDF for relations, etc.
• Leverage best-of-breed open source components from Apache, JBoss, Eclipse, etc.
• Seamless migration from CPS 3: provide tools to automatically migrate configuration and content from CPS 3-based applications to the YellowCake infrastructure (thanks to CPS 3's current configuration infrastructure).

New infrastructure components

We plan to keep all good features and concepts from CPS 3 (CPSSchemas, CPSDocument, CPSRepository, CPSDirectory, CPSWorkflow, overall technical architecture, applicative components, etc.). We are just extending and improving them to embrace industry-standard and existing technologies.

• CPSRepository: a Java Content Repository (JCR) compliant content repository that will enable CPS content to be directly accessed from a JCR level2 compliant API. Relying on the robust Apache Jackrabbit, it will allow us to share maintenance costs of this common pieces of infrastructure with the open-source community, while focusing on extending the platform for the benefit of our customers, users and developers. Note: Nuxeo has joined the JSR-283 (aka JCR 2) expert group.
• Capsule: Capsule is the JCR-wrapping component that enables Zope and CPS to interact with a JCR content repository. Capsule can transparently store and retrieve CPS content objects into Jackrabbit (or any JCR level 2 database). It relies on Jackrabbit for content storage and retrieval.
  Capsule takes care of Zope and Jackrabbit transactions integration (handling XA transactions in Zope). From a technical point of view, Capsule is based on ZODB.Persistence and ZODB.Connection. It acts as a ZODB mount point, that provide transparent access to the JCR.
  To ease the migration and minimize impacts on existing CPS components/applications, CPSSchemas will be kept backward compatible and will load its schemas directly from Jackrabbit. Content types will be defined using XMLSchema and dynamically loaded into Jackrabbit at runtime to create NodeTypeDefinitions. To read more about Capsule technical specs, please consult the SVN repository, for now: nuxeo.capsule and nuxeo.jcr.
• CPSRemoting: bidirectional cross-language integration service that will allow external components to interact and to provide services within the CPS Platform. CPSRemoting exposes an easy-to-use high-level API and distribute CPS events to external event services. It is the bridge between CPS and all other components of your information system. Take any C++, Java, Python, or .NET component, then write a simple interface and plug it into the CPS component registry. That is all you require to extend or to integrate CPS Platform!
• NXLucene and CPSLuceneCatalog: NXLucene is a webservice-enabled indexing server based on Lucene that provides to any application a remote indexing and search service.
  CPSLuceneCatalog is a backward-compatible PortalCatalog/ZCatalog replacement for CPS, relaying on NXLucene. It enables any CPS 3.4 instance to instantly use the power of Lucene as its indexing/search service. Furthermore NXLucene and CPSLuceneCatalog can be used to index several CPS instances (and virtually any application) in the same indexing engine, hence providing unified search across all content-stores. Typical performances on a real-world application (the French law: +190k documents, 50+ fields/document): <5ms to get results from NXLucene and <0.15s to display results in the CPS UI. NXLucene offers XML-RPC and SOAP interfaces (and will soon provide ICE too).
  For more information, see: NXLucene website and source code (NXLucene, nuxeo.lucene, CPSLuceneCatalog).
• CPSMessaging: CPS EventService is one of the key components of the current CPS 3 infrastructure, allowing a global publish/subscribe event system within the CPS platform. For YellowCake, we are replacing it with CPSMessaging, that extend the concept to plug CPS onto an external JMS-compliant message bus (JBoss Messaging or ActiveMQ). It will dramatically eases existing application integration. For example, thanks to CPSMessaging, you can integrate CPS with your JMS-based message bus (or ESB).
• CPSPortlets: JSR-168 compatibility is being added to CPS portlet engine. This will enable CPS to be easily integrated into any JSR-168 compliant portal.
• CPSAudit: a global auditing and logging service, that log all events that occur in the platform into an SQL database. CPSAudit is used, for example, to track content history, workflow actions, and user actions (replacing current CPS mechanisms). CPSAudit is designed to easily create custom reports using any graphical reporting tool (like Eclipse BIRT). CPSAudit can be leverage to create, for example, FDA / SarBox compliant audit reports.

New applicative-level components

Along these core improvements to the CPS infrastructure, new applicative level components are on the way and will be the topic of a future post (give me 5 days :-).
Here is the teaser: reporting, retention management, rich client, physical archive management, Microsoft Office integration, etc.

Join in!

Nuxeo's team and the CPS community would be delighted to get feedback and answer questions.
To get more informations and join the community:

• Design documentation and early code is available from our SVN
• subscribe to the mailing-list to send feedback and discuss.
• CPS Community Portal

We hope to see you there soon!

(Post originally written by Eric Barroca on the old Nuxeo blogs.)

Nuxeo joins JSR-283 and the AIIM's iECM project

Following our goal in opening CPS Platform to industry standards, I am pleased to announce that Nuxeo has joined the JSR-283 (aka Java Content Repository 2) expert group and the iECM project at AIIM.

Florent Guillaume will represent Nuxeo in the JSR-283 expert group. He will bring more than 5 years of experience building content repositories, versionning systems and managing content for large organizations.

The iECM project, hosted by the AIIM, aims at creating a new international standard to ease ECM systems interoperability. The project statement says :

Interoperable Enterprise Content Management (iECM) will create an interoperability framework that enables information sharing across organizational and system boundaries.

I am delighted to represent Nuxeo in this project and work together with all major ECM software vendors to create this new interoperability standard. I really hope it will benefit to our customer and minimize the vendor-locking in this market.
At Nuxeo, we think that Open source is less about source code public availability (still, it's required) than about reducing vendor-lockin for customers benefit. Who cares about your source code if there is no documentation and if all data are locked in your repository?

Interoperability is definitely one of the key benefits of open source!

I cannot wait to see how open source and standards will transform the ECM market (when proprietary vendor will no more be able to use interfaces and content lock-in to keep their customers).

Open competition is on the way, for all ECM customers benefit... And we are working on it! :-)

(Post originally written by Eric Barroca on the old Nuxeo blogs.)

Easter python eggs

It's easter, and only fitting that I play with eggs, so today I have been looking at making python eggs. Eggs are a new way of distributing python software that gets more and more popular. This is mainly because installing it is simple, and because an egg can contain a list of requirements, and the installer will check if these requirements are installed, and if not, it will look them up in the standard Python module library (known as the Cheese Shop), and download and install them.

This means the end of long hours of downloading, compiling and installing all those modules you need to get your python software working. You download just the module you want as an egg and install it, and all requirements will be installed with it, if they have been eggyfied too.

Making eggs is unfortunately a bit tricky. This is because there is no good examples of how to do it, so you have to read through all the documentation, and experiment a lot. This is a shame, because in the end, it turns out to be very easy.

All you need is to install the python module setuptools, and create a a setup.py for your package. Creating it can be slightly tricky, if your package layout is complicated. Also, there is not always clear what the parameters in it should be, and there is quite a lot of searching and trying out (and for my finally some honest debugging of setupttools too figure out what was going on, really).

The parameters are for the most part the same as for making a package with distutils, so if you have a distutils package, all you need to do is to switch out the "from distutils.core import setup" to "from setuptools import setup" and you are done.

Here is an an example of a full setup.py, for the CalCore module:

#!/usr/bin/env python

from setuptools import setup

setup(name='calcore',
   package_dir={'': 'src'},
   packages=['calcore'],
   version='2.0.1',
   install_requires=['zope.interfaces >= 3.0',
            'zope.schema >= 3.0',
            'zope.i18nmessageid >= 3.0'],
   dependency_links=['http://download.zope.org/distribution/',],

   # metadata for upload to PyPI
   author='Martijn Faassen, Infrae;Lennart Regebro, Nuxeo',
   author_email='lregebro@nuxeo.com',
   description='Calendaring system',
   license='GPL2',
   keywords='calendar icalendar',
   url='http://www.cps-project.org/sections/projects/calendar_server ',
   long_description="""CalCore is an advanced, flexible calendaring component for Python. It
    allows the Python developer do write advanced calendaring applications
    either using their own event storage or integrating with external
    calendar servers.
    
    Features of the CalCore include among others:
    
    * Support for making private calendars, shared calendars, resource
    booking and more.
    * integration with iCalendar clients (Apple iCal, Mozilla Sunbird,
    KOrganizer...) using the iCalendar protocol,
    * invitation workflow,
    * meeting support, including helper functions to look for free time,
    * recurring event support (thanks to SchoolTools recurrence
    implementation, http://www.schooltool.org),
    * etc.
    
    CalCore is being used as the core of Nuxeos CalZope and
    CPSSharedCalendar products, products for integrating with Zope and CPS.
    These products provide a complete web-based user interface to the
    CalCore calendaring.""",
   classifiers=['Development Status :: 5 - Production/Stable',
          'Intended Audience :: Developers',
          'License :: OSI Approved :: GNU General Public License (GPL)',
          'Operating System :: OS Independent',
          'Topic :: Office/Business :: Groupware'],
   platforms='All',
   )

You'll note the install_requires parameter, that list the requirements. You can also use ==, of course, and you can list requirements that are only used for testing, and lots of other stuff. Also note the dependency_links parameter, which enables downloads from other places than the Cheeze Shop, in this case Zope.orgs repository of Zope3 eggs.

(Post originally written by Lennart Regebro on the old Nuxeo blogs.)

CPS4/Z3ECM sprint, Final wrapup

The week went very fast, but was very productive indeed, here's a quick wrapup of sprinters work:

Florent & Dario: The boys managed to write a parser to leverage the schemas that exists in JCR into Python structures. They create Zope 3 schemas, rather than CPSSchemas. Everything will be hooked through JackRabbit, so JCR content will become a good citizen for beeing used within Zope 3. They've also created a hierarchy of interfaces for this in Zope 3. All the work is here: nuxeo.jcr

Michael & Lennart: They struggled with Zope 2 & 3 publishers to continue Sidnei's work and came up with an advanced prototype. And, as this was one of the concern, they benched a bit twisted behavior in Zope 2, using funkload and it seem not too slow compared to zserver. The work is on a branch on svn.zope.org.

Jean-Marc: That guy knows everything about Zope 2 and Zope 3 branches now ;). Philipp helped a lot through IRC (thanks) and Jean Marc came up with... wait a minute... a blog entry that resumes it all.

Tarek: After a first release of CPSResourceLibrary, and nuxeo.javascript, I focused on doing tests automation in Javascript. OK that might not be very easy to set up on a buildbot, but on my laptop, it's going to be just perfect to TDD js code in zope.

Joachim: He played with Jean-Marc Cpsskins v3 concepts, by creating some Ajax Portlets in CPS 3.4, relying on CPSResourceLibrary, and nuxeo.javascript. Watch it in CPSAjaxPortlets, it's coming soon babe !

(Post originally written by Tarek Ziadé on the old Nuxeo blogs.)

Open source redemption ?

Last day I was reading a post of Luis Sala from Alfresco and, I have to admit it, I was amazed how he turned to a better self since he met open source.

He seemed to me like vader quitting his mask and seeing the world with his own eyes : stop hiding things to his clients, have fun doing his job, stop selling licences, etc.

Is open source a miracle potion ? I really don’t think so. Nethertheless, working in the open source industry is really amazing. I agree with you on this point Luis. It’s a fantastic approach to satisfy our client’s needs and the best way to produce software. Sadly, open source can not make better personns of us. Personnaly, I prefer let that to psychotherapy.

Be fair and square, that’s how we roll at Nuxeo, since the beginning, and that’s how we build our success. I’m very happy to see that our vision is now shared by people that first laughed at us. Seems like Gandhi’s always right...

(Post originally written by Arnaud Lefèvre on the old Nuxeo blogs.)

CPS4/Z3ECM sprint, day 4, Javascript unit testing automation

For the last day of sprint, I wanted to set up a Javascript unit test automation,
in order to be able to run all Js unitests like Python tests are through buildbot.

Scriptaculous unit testing framework provides a way to do it: the tests results are sent to a url through an Ajax call.

More infos on unit testing javascript here: Scriptaculous Unit testing

Therefore, I have coded:

• a small python script that launches html unit tests with mozilla-firefox, then close it
• a small python http server that retrieves results sent by the javascript

Some caveats to correct before its useable in a Zope project:

• Right now the results sent by the unit test is just a simple 'SUCESS' or 'ERROR' string, and I need to change it so I can get the full error log when they are errors, like the one displayed in the browser, and convert it to Python assertions.
• The ajax call to send results is done from the filesystem html to the localhost. So a trick is needed to make cross-domain ajax posts. The python http server could be used to serve the html tests file as well to avoid this problem, so I would have a complete js test server.
• It needs to work on IE as well, maybe through Wine
  

So to the question "Can I do TDD in JS as smoothly as in Python ?", I'd answer: "Yes, it's coming up :)"

You can check the work here: nuxeo.javascript cps tests

I might try to release the js test runner in a clean package sometimes, as it might be reused.

This entry contains a screencast, you should check it in a browser

(Post originally written by Tarek Ziadé on the old Nuxeo blogs.)

Dump and load foreign subversion repositories using svnadmin and svndumpfilter

What for ?

This is mostly a reminder for mysef about the way of dumping and loading subversion repositories.

Let's say I have two subversion repositories. repoA and repoB. I got a component called P at the root of repoA as below:

repoA /
  P /
    trunk /
    branches /
    tags /

What I'd like todo is to copy P from repoA to repoB preserving all the history logs for P.

Dumping the repository

Let's say I started the developement of P with revision 28000 and my last checkin related to P is at revision 38000. The revision numbers are important because it will decrease the needed time to dump the actual svn repository. Less revisions to dump means less time needed :)

We will use svnadmin to do that.

Go on the server hosting the subversion repository and dump it this way:

$ svnadmin dump --quiet --revision 28000:38000 /home/svn/repoA > repoA.dump

This operation can last a while depending on the size of your repository.

svnadmin dump options :

$ svnadmin dump --help
dump: usage: svnadmin dump REPOS_PATH [-r LOWER[:UPPER] ] [--incremental]

Dump the contents of filesystem to stdout in a 'dumpfile'
portable format, sending feedback to stderr. Dump revisions
LOWER rev through UPPER rev. If no revisions are given, dump all
revision trees. If only LOWER is given, dump that one revision tree.
If --incremental is passed, then the first revision dumped will be
a diff against the previous revision, instead of the usual fulltext.

Valid options:
 -r [--revision] arg   : specify revision number ARG (or X:Y range)
 --incremental      : dump incrementally
 --deltas         : use deltas in dump output
 -q [--quiet]       : no progress (only errors) to stderr

Filtering P component from repoA's dump

What we are interested in here is only component P. We need to filter out for P component. We will use svndumpfilter to do that.

$ cat repoA.dump | svndumpfilter include P > P.dump

svndumpfilter options:

$ svndumpfilter --help
general usage: svndumpfilter SUBCOMMAND [ARGS & OPTIONS ...]
Type 'svndumpfilter help <subcommand>' for help on a specific subcommand.

Available subcommands:
  exclude
  include
  help (?, h)

Loading back P dump to repoB

Now go on the server hosting repoB. Before that, transfer your P.dump to this server. We will use svnadmin to load it back

$ svnadmin load --quiet /home/svn/repoB < P.dump

svnadmin load options:

$ svnadmin load --help
load: usage: svnadmin load REPOS_PATH
Read a 'dumpfile'-formatted stream from stdin, committing
new revisions into the repository's filesystem. If the repository
was previously empty, its UUID will, by default, be changed to the
one specified in the stream. Progress feedback is sent to stdout.
Valid options:
 -q [--quiet]       : no progress (only errors) to stderr
 --ignore-uuid      : ignore any repos UUID found in the stream
 --force-uuid       : set repos UUID to that found in stream, if any
 --use-pre-commit-hook  : call pre-commit hook before committing revisions
 --use-post-commit-hook  : call post-commit hook after committing revisions
 --parent-dir arg     : load at specified directory in repository

Here we are. You just need to remove P from repoA:

$ svn remove http://svn.foo.com/repoA/P

Of course, what would be really cool, instead of the dump/load operations, would be something like that being possible in between foreign repositories:

$ svn move https://svn.foo.com/repoA/P https://svn.foo.com/repoB/

Thanks Florent for the pointers.

(Post originally written by Julien Anguenot on the old Nuxeo blogs.)

Python event in "Le Monde en Tique" book store in Paris the 13th May

We'll continue our Python evangelism work with Stéfane the 13th May, at le Monde en Tique, a book store in Paris.

We'll meet people from 3 to 6 pm, around the "Programmation Python" book to convince more developers to switch to Python and join the community ;)

Web site: http://lmet.com

Address:  6 rue Maître Albert - 75005 PARIS - FRANCE

Access map: Mappy

Tube, RER:
Ligne 10 : Station Maubert - Mutualité
Rer B : Saint Michel - Notre Dame, Sortie : Notre-Dame
Rer C : Cluny - La  Sorbonne, Sortie : Boulevard Saint-Michel /
Boulevard Saint-Germain

(Post originally written by Tarek Ziadé on the old Nuxeo blogs.)