Zope and Python

“Cristiano”, from Faqs.it, has written a comment about our switch from Python to Java in his blog.

Here is my comment about his comment.

Hi Cristiano,

Thanks for your comments about our Java switch. I fully agree with you that a lot of innovations come from the OSS ecosystem, but there are some wrong facts in your remarks that I have to correct, including the following:

• Python, for instance, did actually copy Java’s syntax (and concept) for annotation (called decorators in Python-speak), not the other way around.

• Generics are irrelevant in dynamics languages, hence dynamic languages can’t have “out-innovated” Java on that matter.

• The sentence “Python doesn’t fit in the enterprise-class web apps market” is in part irrelevant for us, because we are not only in the web apps market, but also in the rich client market (with Eclipse RCP).

• Your opposition (in the last sentence) of “Java vs. OSS” is completely disconnected from the reality: Java is GPL now (or will be RSN), Java OSS communities (JBoss, Spring, Apache, Eclipse, Nuxeo…) are among the most active (if not the most active) OSS communities now. And some of these communities have a lot of money at their disposal, because these are projects for building enterprise-class OSS software (or middleware), backed by serious companies like Sun, IBM, Red Hat, or Nuxeo.

IMHO, OSS is now driving, in many ways, the development of Java, and this means that there is no need to oppose Java if you are (like I am) an OSS believer.

Vincent Massol, of Maven and Cargo fame, came to visit the Nuxeo 5 team yesterday to give us some advice on how to best use Maven 2 to build and deploy Nuxeo 5 and ensure the best possible software quality for the product.

So, expect a few changes (= improvements) to the project’s SVN structure, the Nuxeo 5 Maven site and of of course the build system if you are already a developer (if you’re not, I strongly encourage you to consider becoming one.)

And, BTW, we have identified a few features we could add someday to Maven (as a plugin) or Cargo, so expect some contributions from us in the future (expect, but don't count on it before the Nuxeo 5.0 release next month.)

After the OOo 2.0.4 annoucement that provides new extensions improvement such as licencing and new extension .oxt, the extensions team is hard working setting up an ecosystem for developping extensions

The main points actually worked on are

• Translation toolkit : A long discussion has occured focusing mainly on the backend storage format (xml xcu OOo file format, po files, java properties format). While this is important the main focus regarding Extensions usability is to give developpers a cool and simple API to use and translators of every skill level a versatile tool to provide and test translations
  It seems something is beeing studied at Basic IDE level but this rather a new sun internal project and will be limited to dialogs translation on first release. This may be available in OOo 2.2+ and probably not downward compatible.
  As a short term solution, our great Paolo proposed a solution based on pyUNO and xcu file. it is a cool way to go as it will allow extension developpers to have their work being used in OOo 2.x versions and this toolkit can even be enhanced to support other file format.
• About deploying extensions, the actual command line based utility is rather secure but restrictive. It does not allow to use an existing library repository. The problem is that it sometimes behave too silently when de-activating some extensions. Some ideas (use of xlc files) and context restrictions are being discussed to see if something can be improved for user experience
• A new tool demonstrated at OooCon is now available - This NetBeans plugin is really cool and i think that an Eclipse one will also be available :) 
• A small team is setup for the web site - A great Art design work is being started by Ben while Jean-Pierre setups a first prototype. This will be available really soon as an alpha proposal release when server will have required tools
  

if you wish to join the Extensions team, please do. Everybody is welcomed, just tell us, we will find something for you ;)

Speaking about extensions, one as to note the new Sun Java OOo Blogger. This reminds me Caolan's work from last year. The same goal, done and freely available in pyUNO. It is freely available as a study case and use. I bet originality will be for the next extension. But if you want to fund SUN for supporting OOo, it is a way ...

At OOo level, a new team is built for a Creative Commons licence inclusion in OOo documents. If you whish to help visit them on their dedicated mailing list

And for volunteers, I give here some rough ideas for extensions. But, everything else is possible :)

• an impress autostart : OOo does not have a pps-like file format. The idea is to define a DocumentInfo.Info1 parameter, check it though a resident extension on impress file opening and auto start the presentation (level required : easy/medium)
• on-the-fly translation : on the model of OOoWikipedia, retreive the translation of an hilighted word and display it in OOo (level required : easy)
• S5 is a web based light format - create an assistant/filter allowing to export impress files to this format. mainly XSLT stuff but a wizard can also be setup (level required : skilled) 
• logo is a cool language for children to start with. OOoDraw API could be used to embedd such a tool. Note that an engine already exists in python, so using pyUNO could be a good idea
  

These tools creation and how they can be packaged as Extensions can be discussed on users@extensions.openoffice.org. The core part, the api use and help remains on dev@api.openoffce.org mailing list

I propose you to visit our enriched Wiki (some part are even being translated but any mediawiki tool pointer for multilingual support would be great) and join us on dev@extensions.openoffice.org to build the ecosystem, users@extensions.openoffice.org to start building extensions and #ooo-ext freenode IRC channel for a live chat

Nuxeo is switching its ECM to Java, and we're using JCR for our document storage. JCR (Java Content Repository, standardized by JSR-170 and the upcoming JSR-283) is a young specification with a promising future — but what's its point, you may ask, as all existing content management systems are already storing content very well without it? Its goal is interoperability between vendors, which will make it possible for people who write applications needing to store content to have a unified API for such manipulations. All major content repository vendors are active in the JSR-283 expert group, and all are working on JCR bindings for their various proprietary repositories.

Of course a standardized and wildly successful way of manipulating content already existed before JCR: SQL. But SQL and JCR have a different focus:

• SQL is a language; it manipulates rows and is geared toward generic relation manipulation,
• JCR is a Java API; it manipulates nodes and is geared toward hierarchical manipulation (parent-children).

SQL and JCR have quite different underlying data models:

• SQL's model is that of tables with fixed schemas, and relations between tables,
• JCR's model is that of a tree of nodes with flexible schemas and with parent-children relations as the main focus — although other types of relations exist.

JCR also offers higher level features than SQL, notably workspace and version management.

For many kinds of applications, there is a focus on being able to arrange documents in folder hierarchies, and to have a wild variety of structure for these documents. In this case, a storage model based on JCR is much more suited than something based on SQL.

It should be noted that many things in the computing world already are based on the notion of folder hierarchies storing arbitrary documents:

• filesystems are a tree of (unstructured) documents,
• Revision control systems are based on filesystem concepts but add a lot of structure on top of them,
• WebDAV (along with DAV and DELTAV) is a protocol that addresses documents using a path, and where documents have a flexible set of properties,
• most proprietary content management systems are based on (or offer the notion of) organizing the content in a hierarchy,
• Nuxeo's CPS is itself based on this classic model.

This is why JCR has emerged as a common and useful storage API for all these use cases.

For an ECM framework like Nuxeo 5, JCR interoperability can be seen from two different directions:

• JCR provides an API that we can use to store our content, to make us vendor-independent and flexible regarding storage,
• JCR provides an API through which we can expose our content, which makes our platform usable by any external system that understands it — we're ourselves a vendor providing JCR bindings.

For its initial release, Nuxeo 5 is focusing on the use of JCR as its main storage implementation and uses Jackrabbit to store most of our content. In the future, we'll also provide JCR bindings so that the high-level content we provide can be directly accessed from external applications using JCR too.

In a content management system, the actual data that the system or the users manipulate comes from many kinds of sources. Content can come from a JCR repository, or from a relational database, or from an LDAP directory, or from a semantic storage engine like Jena, or from any other kind of open or proprietary storage engine.

But fundamentally all these kinds of content, which I'll call "records", aren't very different:

• a record can be created, viewed, modified, deleted,
• a record can often be copied or moved,
• a record obeys a schema that can be known to the system, this means that its individual fields are strictly typed,
• when being viewed or modified, a record has a user interface that is based on forms, labels, widgets, depending on the schema,
• records can be searched and a result set returned,
• records can be listed in a compact form (search results, folder contents, user dashboard, workflow workitems, RDB table listing, user information browsing, etc.),
• records have an identity (like a unique id) or a location (like a path), sometimes both.

One of the strengths of CPS is to use a common abstraction for many of these concepts, embodied in the CPSSchemas component. In Nuxeo 5 we want to go further and provide even more integration for all these, the base components for these abstractions are NXCore and NXTypeManager.

The reasons to strive for convergence are numerous:

1. this merges into unique concepts things that had been previously separated because of different implementation choices. For instance, an LDAP schema is not fundamentally different from an SQL schema (or from an XML Schema if one is interested in the relevant subset).
2. this gives the programmers a common API for all data-related operations, which means more reusability. For instance changing an attribute in an LDAP entry doesn't have to be different from changing the title of a document or changing a value in an RDB row; processing a list of search results to display them in a table doesn't have to be different from processing the children of a folder to display the folder's contents.
3. this gives the framework developers a way to optimize some operations because of commonalities in the underlying implementations. For instance you don't need three kinds of events dispatching for "LDAP entry modified", "RDB row modified" or "document modified".
4. this gives the users a unified way of manipulating different kinds of data when there's really no need to have a different UI for them. When a user fills a form it's really the same process whether he's modifying his personal preferences, adding a keyword to a document, or changing a quantity in an RDB row.
5. this allows very simple migrations between storage technologies, when these are felt necessary. A customer could start with an LDAP database for its user base and later have the need to move them to an RDB table. User entries in an RDB table may need to be versioned and moved to a JCR storage. An application should survive all these with only configuration changes and no code to rewrite.

It should be noted that this means that JCR is in no way the primary storage model for Nuxeo 5, it's only the first one to be implemented. In the future, it will be possible to store documents in LDAP or an RDB. When a suitable storage model is devised and implemented, you'll be able to apply workflow or versioning to RDB-based documents for instance.

This convergence is quite exciting to us, and our goal is to allow people to build complex applications with Nuxeo 5 in a more straightforward manner.

Just to let you know: I have decided to split this blog in two distinct blogs for more clarity, ecm, java and cps related things will remain here, while python related things will now be written in a secondary blog: Carpet Python.

Euriware and Nuxeo give you a good reason to wake up early in the morning ! We'll have the great pleasure to welcome you next thursday (10/19/2006) from 8 to 11 AM for an information breakfast, to share our experience about Open Source ECM projects and document management.

This meeting will give you the keys to transform your Open Source ECM project into a real success and to optimize the use of Open Source softwares.

I'll be pleased to present Nuxeo's document management products (including a live demonstration) and to discuss with you about the advantages of Open source ECM solutions.

And last but not least... As a cherry on top of the cake, Areva NC will be there to make a testimonial about their own success based on Nuxeo's solutions.

Be there and bring friends !

We announced our switch to Java two weeks ago. There was of course a lot of preparation before the announcement (besides just coding). That included: creating the new Nuxeo.org website, writing a FAQ, preparing slides, writing announcements for various media (InfoQ, TSS,…), and anticipating some discontent in the Zope community.

Well, two week after going public with the announcement, I’m glad to say that we are completely pleased with the way things went.

We already have released one of the key components of the platform, Nuxeo Runtime and are preparing to release Nuxeo Core this week. Overall, work on the project is going on at a steady pace and we are on track to meet the next milestones of our roadmap.

Our announcement got noticed both by the main Java sites, and countless blogs (including some in languages that we don’t understand without Google Translator).

There has been a little discontent, as expected, in the Zope community, but we mostly got some nice messages of people saying they understood our choice. The most heated discussions took place after Jean-Marc’s post (and its followups) supporting our change.

And, most importantly, we got many subscriptions to the mailing list, which shows there is a real interest in the developer community.

On the business side, we’ve had many calls from system integrators.

I’m especially glad to notice that in two weeks, we got contacts from several new systems integrators, telling us that, now we are using Java, they will be very happy to work with us on new projects. Consequence: we now have a commercial contact with 8 of the 10 leaders of ECM integration in France (some of them the french subsidies of international groups), as well as interesting partnership projects with a big name (I mean BIIIIG) software vendor (more on this another day).

I gotta admit I was wrong...

I recently tried out Pydev and Pydev extensions on Eclipse 3.2 for my
Python and Zope developments. Wouah !

I've been using these plugins for couple of weeks and I already can't
switch to emacs anymore. I wouldn't have believed It could have ever
happened. I've been using emacs for years and somehow refused to try
out any IDE for Python because I thought it wasn't as crucial for
Python development as for other languages such as Java, C#, C++
etc....

I love this feeling. I mean when you find out a tool and are wondering
how you've been living without it for so long that you can't go back to
the old one :) I'm in this case with emacs. I simply can't
anymore. (Well, I could but you see what I mean)

Pydev really brings a lot.  I was surprised by the progresses made by
Fabio on the Pydev front and especially what Pydev extensions bring
related to code completion.

I let you check out the screencasts to get a quick overview if you
don't have time to install and play with it (or you are not convinced)

For Zope development there are some things I'm missing that require
some command lines beside Pydev. (Or these are maybe some things I
didn't find since I'm relatively new in Pydev development)

•  Support for zope.interface

      This is a bit annoying since pydev doesn't recognize
      zope.interface and thus shows errors on those classes. I might
      even dream about a full integration checking the Python classes
      implementing interfaces such as in Java. And yeah, I'm still
      dreaming about builtin Python interfaces but eh wait ... Python
      guys don't want that...I need to stop dreaming...

• Support for doctests

      Pydev doesn't support doctests. Although, you can run external
      commands from Eclipse. I'm using them to specify a zope.testing
      based tests.py as external command and I get back the result in
      the eclipse console. It would be a plus if it could directly run
      them from the Eclipse as doctests as the other Pyunit tests.

•  Python specific package browser

      Here, I'm especially talking about the refactoring. Move and
      rename you can find in the Java package manager. So that you can
      rename module names and don't worry about changing the references
      in other Python projects in the workspace. Or still move modules
      with the same behavior. It really does ease the refactoring in
      Java.

• Zope server plugin
  
  
• Debugger integrated with Zope
  

To sump up, Pydev and Pydev extensions rock. It's really promising and
it addresses a huge lack related to Python development.

Please don't tell me about other Python specific IDEs since this is
not what integrators are expecting. They already do have Eclipse on
theirs boxes all over the place so they just want a new Eclipse
plugin. (and they are right) Another industry reality. I guess IDE
war is over ... as well.

I guess Éric is going to like this post ;)

The word "triage" in programming has been around for a while, but seem to have gained popularity lately. What it is, is basically a new word for "bugprioritizing", being easier to say and spell. But if we look at what the word comes from, we see that behind this lies a concept that is often forgotten in the usage.

"Triage" comes from french and means sorting, so nothing new there. Howerver, it comes to the programming world from the world of emergency health care, where it is used to allocate health care resources in emergencies, when there are more people needing care than there are health care available. In computers then, it becomes a system of allocating programming time to bugs, when there are not enough programmers to fix all the bugs.

How should this be done? Well, first of all, we of course need to take a look at how serious the bug is. What categories you have depend largely on what bugtracking system you use, and it's also a matter of personal taste. Personally, I would prefer these categories:

• A blocking/critical bug is a bug that means the system or part of the system gets unusable. It has no workarounds, and affects most users of the system.
• A serious/medium bug is a bug that is a big problem for many users and has no workaround, or affects everybody, but has a workaround.
• A minor bug is a bug that affects few users and has a workaround, or has no workaround but only is a problem in very extreme edge usercases.
• A trivial bug is not an actual problem for anybody. Could be spelling errors or ugly design.

Bugs also usually have a priority field. Prioritizing or triaging a bug is usually just setting this field. But triage is not prioritizing, it's resource allocation. And although in medicine, the resources are helath care workers, and the criticality is time, in programming, the resurce is programming time, and the ciriticality is rather the knowledge to fix the really difficult bugs.

That's OK as long as you work in a company, because there you can take the bugs allocated to you, in the order your boss prioritized them. If you get problems, you ask your co-workers. But in the open source world there is bigger differences. Also in the open source world, people assign bugs to themselves, and if you think a bug is over your head, you will simply never try to fix it.

Therefore, the bugs importance should not be the only data you tag the bug with, but also how hard the bug is to fix. Most bugtrackers only have severity and priority. Some, like JIRA, has a field where you can estimate it. Although that's more of a XP project management features than a help in triage, it can be used as such. If you have an hour over to bugfix, there is no point in trying to understand a bug who is estimated to take hours to fix.

But what would be even more helpful in an open source situation would be a difficulty field. The experienced bugfixers can look through the bugs, note severity and difficuly, and then not fix the easy bugs. It's tempting to fix the easy ones because quick fixes are satisfying and you feel productive. But if the wanna-be contributor can fix bugs marked as "easy" and understand and fix them, then we have not only gotten a new contributor, we have saved time for the experts.