Nuxeo Developers Blog

Introducing fise, the Open Source RESTful Semantic Engine

As a member of the IKS european project Nuxeo contributes to the development of an Open Source software project named fise whose goal is to help bring new and trendy semantic features to CMS by giving developers a stack of reusable HTTP semantic services to build upon.

As such concepts might be new to some readers, the first part of this blog post is presented as a Q&A. 

What is a Semantic Engine?

A semantic engine is a software component that extracts the meaning of a electronic document to organize it as partially structured knowledge and not just as a piece of unstructured text content.

Current semantic engines can typically:

• categorize documents (is this document written in English, Spanish, Chinese? is this an article that should be filed under the  Business, Lifestyle, Technology categories? ...);

• suggest meaningful tags from a controlled taxonomy and assert there relative importance with respect to the text content of the document;

• find related documents in the local database or on the web;

• extract and recognize mentions of known entities such as famous people, organizations, places, books, movies, genes, ... and link the document to there knowledge base entries (like a biography for a famous person);

• detect yet unknown entities of the same afore mentioned types to enrich the knowledge base;

• extract knowledge assertions that are present in the text to fill up a knowledge base along with a reference to trace the origin of the assertion. Examples of such assertions could be the fact that a company is buying another along with the amount of the transaction, the release date of a movie, the new club of a football player...

During the last couple of years, many such engines have been made available through web-based API such as Open Calais, Zemanta and Evri just to name a few. However to our knowledge there aren't many such engines distributed under an Open Source license to be used offline, on your private IT infrastructure with your sensitive data.

Why would I want to semantically annotate my content?

Linking content items to semantic entities and topics that are defined in open universal databases (such as DBpedia, freebase or the NY Times database) allows for many content driven applications like online websites or private intranets to share a common conceptual frame and improve findability and interoperability.

Publishers can leverage such technologies to build automatically updated entity hubs that aggregate resources of different types (documents, calendar events, persons, organizations, ...) that are related to a given semantic entity identified by an disambiguated universal identifiers that span all applications.

I you are not yet convinced please have look at this BBC use case and this 3 minutes video by the fine freebase folks.

How to use fise?

Setting up a fise instance

You can test fise using the online demo or you can download a snapshot of the all-in-one executable jar launcher (67MB) or you can build your own instance from source. If you want to run your local instance just launch it with a java 6 virtual machine as follows:

java -jar eu.iksproject.fise.launchers.sling-0.9-SNAPSHOT.jar

And point your browser to http://localhost:8080 instead of http://fise.demo.nuxeo.com in the following examples.

Overview of the fise web interface

Once the server is up and running, fise offers three HTTP endpoints: the engines, the store and the sparql endpoint:

• the /engines endpoint all the user to analyse English text content and send back the results of the analysis without storing anything on the server: this is stateless HTTP service

• the /store endpoint does the same analysis but furthermore store the results on the fise server: this a stateful HTTP service. Analysis results are then available for later browsing.

• the /sparql endpoint provide a machine level access to perform complex graph queries the enhancements extracted on content items sent to the /store endpoint.

The engines endpoint

Let us focus on the /engines endpoint. The view first list the active registered analysis components and then ask for a user input. Type an English sentence that mentions famous or non famous people, organizations and places such as countries and cities. I your are lazy, just copy and paste some article from a public news feed such as wikinews and submit your content with "Run engines". Depending on the registered engines and the length of your content, the processing time will typically vary from less than one second to around a minute.

Submitting text content to the /engines endpoint using the web interface

By default fise launches three engines in turns:

• the first engine performs named entity detection using the OpenNLP library: it will try find occurrences of names of people, places and organizations based on a statistical model of the structure of English sentences

• the second engine tries to recognize the previously detected entities using a local Apache Lucene index of the top 10,000 most famous entities from DBpedia. This index is configurable and will be improved in future versions of fise.

• the last engine then asynchronously fetches additional data from DBpedia such as the GPS coordinates of places, thumbnails and Wikipedia descriptions of the recognized entities. Fetched entities are cached in the fise store for faster lookup next time the entity is recognized. A summary of those informations are then display the results in the fise UI as columns of entities and a word map display the locations:

 

Overview of the extracted entities in the submitted text.

Using the REST API

Up until now we have used the web user interface for human beings who want to test the capabilities of the engines manually and navigate through the results using there browser. This is primarily a demo mode.

The second way to use fise is the RESTful API for machines (e.g. third party ECM applications such as Nuxeo DM and Nuxeo DAM) that will use fise as an HTTP service to enhance the content of there documents. The detailed documentation if the REST API is available on a per-endpoint basis in the Web UI by clicking  on the "REST API" link in the top right corner of the page:

 

Accessing the inline documentation for the REST API

Here is a sample call to the engines endpoint REST interface using the curl command line tool:

curl -X POST -H "Accept: text/turtle" -H "Content-type: text/plain" \
 --data "Fise can detect famous cities such as Paris." \
 http://fise.demo.nuxeo.com/engines/

Please note that the output of this call will be formatted as text/turtle (a standard RDF serialization format). Again this is not meant to be consumed by regular human beings but only by machines (or semantic web engineers). The list of available serialization formats is detailed in the inline documentation.

The road ahead

To close this first blog post, here are a couple of un-prioritarized improvements I would like to implement in the coming months.

Multi-lingual support

Right now the packaged engines can only deal with English text content. We plan to progressively add statistical models for other languages as well.

Relations extraction

Right now if you submit a sentence that starts with "United Kingdom prime minister David Cameron declared to the press..." you will get an output such as:

"David Cameron" is detected as a person but not recognized since the fise index was built on a DBpedia dump extracted before his election. Furthermore fise is currently not able to extract the relation between the entity "David Cameron" and the entity "United Kingdom". In future versions of fise we plan to extract the role "prime minister" that links the person to the country. This should be achievable by combining syntactic parsing with semantic alignment of english words with an ontology such as DBpedia.

Extracting relations between entities will help knowledge workers incrementally build large knowledge bases at a low cost. For instance, this can be very interesting for economic intelligence or data-driven journalism: imagine automatically building the social networks of public figures from news feed and their relationships with business entities such as companies and financial institutions for instance.

Integration with Nuxeo EP

Right now fise is a standalone HTTP service with a basic web interface mainly used for demo purposes. To make it really useful some work is needed to integrate it with the Nuxeo platform so that Nuxeo DM, Nuxeo DAM and Nuxeo CMF users will benefit from a seamless semantic experience.

Content views: nice and easy configuration of documents listings

Let's talk about designing a Web UI framework, to use big words :)

Sometimes it's difficult to assess the configurability needs for a framework, and sometimes these needs change and have to evolve. Ideally, the new design will allow to answer to most common needs with little work, but will keep it possible to achieve more complex tasks. It will also take care of restrictions that the underlying frameworks impose.

This is why we decided to introduce a new concept in the Nuxeo UI Framework: content views.

A content view is a notion to define all the elements needed to get a list of items and perform their rendering. The most obvious use case is the listing of a folderish document content, where we would like to be able to:

• define the NXQL query that will be used to retrieve the documents, filtering some of them (documents in the trash for instance)
• pass on contextual parameters to the query (the current container identifier)
• define a filtering form to refine the query
• define what columns will be used for the rendering of the list, and how to display their content
• handle selection of documents, and actions available when selecting them (copy, paste, delete...)
• handle sorting and pagination
• handle caching, and refresh of this cache when a document is created, deleted, modified...

The content view framework aims at putting together several configurations that were already available in Nuxeo and have become complicated to maintain and to configure.

• query models: XML configuration that makes it possible to define a query, its needed parameters, and that can use a document model to retrieve query parameters (typically from a search form)
• result providers: Java classes that handle pagination of a list of documents retrieved from a NXQL query to a core session
• result provider farms: Java classes (typically Seam components) that handle resolution of contextual query parameters to pass on to the query models
• result provider cache: Java class handling caching and invalidation of result providers
• select datamodel: Java class encapsulating a list of items to handle selection in a JSF environment
• Seam remoting and Javascript methods to handle re-rendering of available actions on selection
• XHTML pages to handle rendering of needed columns, pagination, sorting...

So it had become overly complicated to define a custom listing of items, and most developers eventually got lost when introducing result provider farms. Moreover, Nuxeo Studio could not provide that kind of configurability because it only handles XML configuration or predefined XHTML pages.

Content views make it possible to answer to all these needs, and more, with the following stategy:

• handle rendering of listings using layouts: layouts are configured only using XML configuration and are reusable in different contexts
• improve providers, making them able to handle any kind of item type (not only documents), handle item selection, pagination, multiple sorting...
• make it possible to define the provider, its query and its parameters through XML configuration, using EL expressions to resolve contextual parameters
• configure caching using cache keys that can be resolved using EL expressions, and invalidation using event names
• make it possible to define one or more content views on a document type, but keep them independant from it so that it can be used also for advanced forms

Here is a typical content view definition, used when rendering a folderish document content:

<contentView name="document_content">

  <coreQueryPageProvider>
    <property name="coreSession">#{documentManager}</property>
    <pattern>
      SELECT * FROM Document WHERE ecm:parentId = ?
      AND ecm:isCheckedInVersion = 0
      AND ecm:mixinType != 'HiddenInNavigation'
      AND ecm:currentLifeCycleState != 'deleted'
    </pattern>
    <parameter>#{currentDocument.id}</parameter>
    <sort column="dc:title" ascending="true" />
    <pageSize>20</pageSize>
  </coreQueryPageProvider>

  <cacheKey>#{currentDocument.id}</cacheKey>
  <cacheSize>10</cacheSize>
  <refresh>
    <event>documentChanged</event>
    <event>documentChildrenChanged</event>
  </refresh>

  <resultLayouts>
    <layout name="document_listing_ajax"
      title="document_listing"
      translateTitle="true"
      iconPath="/icons/document_listing_icon.png" />
    <layout name="document_listing_ajax_compact_2_columns"
      title="document_listing_compact_2_columns"
      translateTitle="true"
      iconPath="/icons/document_listing_compact_2_columns_icon.png" />
    <layout name="document_listing_ajax_icon_2_columns"
      title="document_listing_icon_2_columns"
      translateTitle="true"
      iconPath="/icons/document_listing_icon_2_columns_icon.png" />
  </resultLayouts>

  <selectionList>CURRENT_SELECTION</selectionList>
  <actions category="CURRENT_SELECTION_LIST" />

</contentView>

Here the core query page provider defines its contextual properties to perform the query on the core session, and to fill the parameter of its query (the current document identifier). It also states a sort column, and the default page size (20 items).

The cache key is also linked to the current document identifier: this is useful to keep the current page, sorting information, current result layout, etc... in place, until we need to refresh it because some documents may have appeared in the application. The cache size is 10: the cache will be effective until 10 different documents have generated this content view. When this content view is computed for the 11th document, the cache for the first document is lost and will need to be recomputed.

The last part of the configuration makes it possible to define the different listing layouts available for this content view, in case you'd like to switch between different presentations. The selection list and actions handle document selection, as well as buttons available on selection. The re-rendering of available actions is done through ajax calls configured in the layout templates and widgets, because the seam remoting mechanism imposed too many restrictions on the HTML structure of action buttons.

The caching mechanism and rendering are handled by generic Seam components and XHTML templates that will use the defined content view. It is also possible to attach content views to documents, so that their default page displays the given content views provided a category. For instance, here is the configuration of the Folder document type:

<type id="Folder">
  <label>Folder</label>
  ...
  <contentViews category="content">
    <contentView>document_content</contentView>
  </contentViews>
  <contentViews category="trash_content">
    <contentView>document_trash_content</contentView>
  </contentViews>
</type>

Next step is to make Nuxeo Studio benefit from these changes: folderish documents content, search form and results, virtual navigation results will be migrated to content views, so that all its features can be configured using a nice interface.

The complete documentation for content views is available here: https://doc.nuxeo.com/display/NXDOC/Content+views

Nuxeo DAM 1.1 - What you should know about configuration

We just announced the release of Nuxeo DAM 1.1, with some compelling new features such as video storyboarding, video streaming, bulk import and edit, and repository sharing with Nuxeo DM.

This new version also offers some improvements to the some of the application configuration components.

New configuration system (from Nuxeo DM 5.3.2)

A limitation of the configuration system in Nuxeo 5.3.1 and prior versions was that each platform upgrade needed the system administrator to reconfigure the data sources and other system-level configuration files.

We have listened to our own sys admin (who was complaining about that when upgrading Nuxeo Cloud instances) and have changed the way configuration resources are handled.

Basically, the software packages (EAR or WAR for DM, DAM, CMF ...) don't include anymore resources for system level configuration.

These configuration files are now handled by a template system that:

• allows you to easily change configuration (for example change the storage configuration from a given database brand to another one);

• allows for easy upgrades (since the new EAR or WAR won't overwrite your custom config).

One of the direct advantages of this new system is that you can now very easily switch from the default embedded storage to a production-ready storage (for instance, PostgreSQL or Oracle).

You can find a detailed description of the new configuration system here.

Upgrade Notes

Packaging

The packaging system is basically the same as the one used in Nuxeo DAM 1.0.

The only change that may have an impact involves resources that are now managed by the new template system.

This means that resources are no longer embedded inside the EAR but handled in a separated templates directory.

This makes changing configurations easier (like switching from H2 to PostgreSQL) and will also allow for upgrades without having to redo all custom system configurations.

The packaging system is the same for Nuxeo DAM and Nuxeo DM, you can refer to the documentation on Nuxeo DM:

• Installation Guide

• Description of the new configuration system

You can also refer to DAM Installation Guide.

OpenOffice Configuration

We have changed the way Nuxeo starts OpenOffice.

This is an intermediate solution before we upgrade to JODConverter 3.

The new OOolauncher (that replaces OOodeamon) should:

• be more stable.
• be easier to set up (removed the dependencies on JNI UNO libs).

To find out more ....

For the JIRA Release Notes, see Nuxeo DAM 1.1 JIRA Change Logs.

For more information about Nuxeo DAM 1.1:

• Nuxeo DAM - Documentation Center
• Nuxeo DAM - Developer Center

We hope you'll enjoy the release and look forward to your feedback on Nuxeo DAM's discussion forum!

The Nuxeo DAM team.

Nuxeo DM 5.3.2 is available

Summary

The Nuxeo development team is happy to announce the release of Nuxeo EP and DM 5.3.2, the open source ECM Platform and Document Management application.

As you can guess from the version number, this is a maintenance release that includes mainly bug fixes, but also numerous improvements, and some refactoring. Some of these improvements provide the power behind the upcoming new versions of Nuxeo Case Management Framework (Nuxeo CMF 1.1) and Nuxeo Digital Asset Management (Nuxeo DAM 1.1) that are built upon Nuxeo EP and DM.

Nevertheless, more than 300 Jira tickets have been resolved between Nuxeo DM 5.3.1 (that was released exactly 4 months ago) and 5.3.2.

The main new features of Nuxeo DM 5.3.2 include:

• a new configuration system (to make upgrade and administration easier);
• a new Tag Service implementation (complete integration with VCS backend);
• content automation integration;
• VCS improvements (better Oracle support and optimizations);
• new packaging options and installers;
• more translations.

For a more detailed description of these improvements, please see below.

You can download the packages for Windows, Linux, Mac OS on the download page on nuxeo.com.

The upgrade from 5.3.1 to 5.3.2 should be easy, see the upgrade notes below for more details.

Detailed release content

New configuration system

A limitation of the configuration system in Nuxeo 5.3.1 and prior versions was that each platform upgrade needed the system administrator to reconfigure the data sources and other system-level configuration files.

We have listened to our own sys admin (who was complaining about that when upgrading Nuxeo Cloud instances) and have changed the way configuration resources are handled.

Basically, the software packages (EAR or WAR for DM, DAM, CMF ...) don't include anymore resources for system level configuration.

These configuration files are now handled by a template system that:

• allows you to easily change configuration (for example change the storage configuration from a given database brand to another one);

• allows for easy upgrades (since the new EAR or WAR won't overwrite your custom config).

One of the direct advantages of this new system is that you can now very easily switch from the default embedded storage to a production-ready storage (for instance, PostgreSQL or Oracle).

You can find a detailed description of the new configuration system here.

New Tag Service implementation

The Tag Service has been completely integrated with the VCS (repository) core.

As a side effect, this means that Nuxeo Core can now handle low level relations and this opens the way for more efficient queries on the Tags and Relations.

These changes will be leveraged in the 5.4 release of DM.

Content Automation

Content Automation is now part of the Nuxeo DM default distribution.

Content Automation provides a high level API for operations that can be used:

• from Java code

• from HTTP reuqests (via the built-in REST API)

• from Nuxeo Studio Content Automation Chains

  
  

You can find the related documentation here.

Repository (VCS) improvements

Based on extensive benchmarks we did with 10s of millions and 100s of millions of documents in one repository, we've introduced new low level optimizations inside our VCS repository implementation.

We've also worked to improve Oracle database support and have ported our security optimizations (already available on PostgeSQL) to Oracle.

Some of the VCS improvements won't be fully visible until we ship version 5.4 in Q4 2010, but the infrastructure work has been done and this opens the way for new features in the next release.

Packaging and installers

We have also worked on improving the way Nuxeo DM is delivered to the end users that want to test the platform.

These improvements includes:

• a new MS Windows installer including a small monitor and new Nuxeo control panel;

  
  

• Debian and Ubuntu packages;

• a ready to run virtual machine image (to run Nuxeo DM in VirtualBox or VMWare).

More translations

Thanks to the help of our community, we have a new translation to Polish (by Adam Lozy), as well as many improvements to the translations we had before: Arabic, Basque, Catalan, Chinese, English, French, Galician, German, Italian, Japanese, Polish, Portuguese, Brazilian, Russian, Spanish and Vietnamese.

Upgrade Notes

Code migration

5.3.2 is fully backward compatible with 5.3.1 (no compat package is needed).

So, you should have no issues with running your custom code against 5.3.2. If you have any problems, you can contact Nuxeo Support.

Packaging

The packaging system is basically the same as the one used in 5.3.1.

The only change that may have an impact involves resources that are now managed by the new template system.

This means that resources are no longer embedded inside the EAR but handled in a separated templates directory.

This makes changing configurations easier (like switching from H2 to PostgreSQL) and will also allow for upgrades without having to redo all custom system configurations.

The documentation has been updated accordingly:

• Installation Guide

• Description of the new configuration system

Data

The only changes done between 5.3.1 and 5.3.2 are the way tags are stored.

Because the Tag Service is now directly part of VCS, some small changes have been done.

Nevertheless, migration should be automatic and transparent.

If you have any problems, you can contact Nuxeo Support.

Configuration

We have changed the way Nuxeo starts OpenOffice.

This is an intermediate solution before we upgrade to JODConverter 3.

The new OOolauncher (that replaces OOodeamon) should:

• be more stable.
• be easier to set up (removed the dependencies on JNI UNO libs).

Nuxeo Studio 1.3 has been released

The goal for this release was to extend the scope of the capabilities of configuration with Nuxeo Studio - and to ensure the usability of Nuxeo Studio to meet your integration projects.

We are happy with the results and are looking forward to getting your feedback! You can now implement document workflows, add custom traceability in any automation chain, let the user download files you created in chains, leverage the vocabulary labels to automatically build title of documents or update whatever metadata. Furthermore, starting from the soon-to-be released Nuxeo DM 5.3.2, every automation chain you do in Nuxeo Studio will be exposed as a REST webservice - and Nuxeo Studio allows you to filter the access to those webservices. Finally we did a lot regarding the usability and industrialization of the Nuxeo Studio development process: Versioning control of your configuration project, project version tagging, configuration project history, integration with Maven...

For testing this release, I used the scenario of tender management: implementing features such as worklow validation for tender pieces, publishing of the tender, downloading of all the published tender pieces of documents at one time, filling audit table for statics... Even though this is the third release I've worked on, I am impressed and surprised of how quickly you get the configuration done! Many thanks to the Nuxeo Studio team, which is already on track for the 1.4 shot, with exciting new features - can't wait to tell you more about it, stay tuned!

While the official release note is available - release note contains full details -we provide some key highlights and items to note here:

Main new operations:

• Create task: You can now create and assign tasks to users from an operation chain. The "create task" operation is very configurable: you can choose which date, which comment, which directive you want, and also the assignment behavior. You can also compute, in the automation chain, the list of persons you want to assign the task too. Last but not least, you can bind automation chains to accept or reject events, thus designing a complete document workflow process.

• Get users and groups: This new operation resolves all the users and groups having a given permission on a document. This is very useful, in conjunction with the previous "Create task" operation.

• Get Principal emails: This operation returns emails of users having a given permission on a document. Can be used in conjunction with the "Send email" operation to send custom notifications.

• Log Event in Audit: Thanks to this operation, you can now feed audit trail with custom events. If you provide a particular validation chain, you can add this operation in the end so that you add an entry in the document history, to log the execution of this particular chain.

• Download file: This user interface operation enables you to provide buttons whose effect is to download a file in the end, that file containing a specific document, or a zip of a document selection, or a pdf transformation, or whatever you need!

Rest binding for Automation chains

Operations and automation chains are now bound to a REST API. As a direct result, you can use Nuxeo Studio to implement a custom REST API for you project - with not even one line of java code! See About the automation rest service for more information on using those webservices.

Better usability of Nuxeo Studio in integration projects

• Projects are now under source control system: We integrated Git so that it is possible to commit configuration states, restore previous configuration states, tag specific versions of your project, be able to realize difference between versions and see history of the configuration. You will now be able to control what you sent in qualification, what is in development, etc... Also, it will make it easier to work as a team around a Nuxeo Studio project, as you can easily follow what happens.

• Nuxeo Studio now integrates with Maven. If your project needs side eclipse development in addition to your Nuxeo Studio configuration, you can declare the Nuxeo Studio plugin as a dependency of your global development project, using Maven:

  <dependency> <groupId>nuxeo-studio</groupId> <artifactId>myproject</artifactId> <version>1.0</version> </dependency> </code>

  The "1.0" version corresponds to a tag on your Nuxeo Studio project. When building your project, Maven will automatically download from the Nuxeo Connect platform your customization.

• You can leverage System properties, for instance, to differentiate values depending on whether it is a dev, qualification or production platform. Nuxeo EP allows system properties values, which are properties defined in "properties files" in the config folder of your Nuxeo instance. A sample property file is the "notification.properties" where you store the url of your server to include it in email notifications. Now all those properties are loaded in Content automation context, and thus can be used in scripting instructions.

Various improvement/fixes:

Filter elements by path, make the "summary tab" visible for folderish types, make the "create documents " appear, be able to translate a vocabulary ID in its label (thanks to a scripting function), and make it possible to configure the fixed part of the advanced search...

Useful links about Studio:

• Studio trial

• Studio documentation center

• About the automation rest service

• Follow evolution of Studio project and be alerted of coming new features

• Studio users forum

Nuxeo EP & Nuxeo DM 5.3.1 available!

We are happy to announce a new release of our ECM platform, Nuxeo EP, and the collaborative document management application built with the platform, Nuxeo DM. The version number for the platform and for Nuxeo DM is the same, 5.3.1. It is available today and we encourage all our customers and partners to upgrade. We encourage all our customers and partners to upgrade.

Downloads are available on nuxeo.com download area and through the Nuxeo Connect portal for our customers.

This new release mainly brings improvements and bug fixes to the software. We have improved existing components and services, but very few new API have been added. This new version is fully backward compatible, hence upgrade is painless and requires no data migration or code change (see upgrade notes below).

Major improvements:

• The CMIS connector is now included by default.

• OpenSocial support has been improved and upgraded (with OpenSocial 0.9 and OAuth support); more compatibility testing.

• The default user interface is more configurable thanks to generalized usage of Layouts (and Nuxeo Studio).

• Our storage engine (VCS) has been improved and optimized for higher data volumes.

See below for a more detailed description of this release's content.

Detailed release content

Security / UserManager

Computed group support

Default UserManager implementation now supports computed virtual groups.

Superuser configuration

The configuration to define the "administrator" global group (i.e.: role) has been extended.

You can now easily:

• define your own "administrators" group

• define several single Administrator users

JSF UI Framework

Layouts (forms/views)

The layout system is now used for:

• document listings

• summary screen

This makes the JSF UI more easily configurable.

As a side effect, Nuxeo Studio will now be able to let you customize document listing and summary screen content.

Widgets

More UI widgets have been packaged to be used in forms / views (ex: tags, comments, relations, user suggestion, etc.).

WebEngine

Jersey alignment

WebEngine is now based on Jersey version 1.1.5.

This provides better compliance with the JAX-RS standard and also allows deploying pure JAX-RS applications on top of Nuxeo.

Test framework

Thanks to the new test Framework included in Nuxeo Runtime (Junit 4 / Guice / WebDriver), you can now unit test Web Engine applications. See below for more details.

Storage / VCS

VCS is our document storage engine based on a relational database back-end.

Improved storage adapter for Oracle DB

Several small issues with VCS and hibernate have been solved to improve Oracle support. Improved performances thanks to some low-level optimizations.

Extended cluster support for VCS

VCS clustering is now available on all DB backends: H2, PostgreSQL, Oracle, MySQL, MS-SQL Server.

Query improvements

The NXQL query language has now support for:

• ordered folders

• ILike

• NOT BETWEEN (bug fix)

Default indexing configuration

The default indexing configuration now leverages the supports for multiple full-text indexes.

• one index for title

• one index for description

• one index for all text content

This makes the advanced search screen more powerful.

CMIS

Nuxeo has been working both on the specification effort on the CMIS (Content Management Interoperability Standard) standard, on a Java library under the auspices of the Apache Software Foundation (project Chemistry), and on the implementation of CMIS on top of a Nuxeo content application.

You can find more information on CMIS support on this wiki page.

Chemistry

Chemistry and the nuxeo-chemistry module now provide a better support for CMIS 1.0 in Nuxeo:

• getDescendants and getFolderTree are now advertised, deleteTree works

• allowable actions are now provided

• regular support of Nuxeo documents that are in the trash (they are hidden from CMIS)

• content streams are readable and writable for all documents that support them

• CMISQL:

  • JOIN support has been improved

  • SELECT * now follows the spec

  • CONTAINS, IN_TREE and IN_FOLDER are now allowed

  • CMISQL can now easily be used from Java code in a Nuxeo application

• partial SOAP server support with UsernameToken authentication

• many bug fixes

CMIS support in Nuxeo DM

Nuxeo DM now comes with the CMIS connector by default.

OpenSocial / Dashboard

Shindig 1.1 and OpenSocial 0.9

Nuxeo OpenSocial integration is now based on Apache Shindig 1.1 that provides OpenSocial 0.9 standard support.

New gadgets in the Dashboard

The Nuxeo Dashboard now provides additionnal gadgets that can be used to customize your dashboard.

OAuth support

Nuxeo is now an OAuth consumer: this means you can have a gadgets that consume data from an external server supporting OAuth standard.

This has been successfully tested agains Atlassian Confluence 3 and Jira 4.

See [here] for more details.

Packaging

5.3.1 comes with new distributions profiles:

• Nuxeo DM (including CMIS) on JBoss 4.2.

• "Core CMIS Server" (a lightweight Tomcat packaging with Nuxeo Core and the CMIS connector).

• Nuxeo DM on Tomcat with transaction management and pooling support (aka JTA/JCA).

We have generalized the use of nuxeo-distribution-tools which combines Ant flexibility and simplicity with power of Maven artifacts management.

This simplified packaging system is based on Ant XML syntax and usable either as a Maven plugin, as an Ant task, or as a standalone command.

See here and here for more details.

Test framework

The Nuxeo Runtime now provides an enhanced testing framework that leverages Junit 4, Guice, WebDriver and customizable test annotations.

This makes building complex test cases easier since you can now inject features and configuration into the unit tests.

See here for more details.

Upgrade notes

Code migration

5.3.1 is fully backward compatible with 5.3 (no additional compatibility package is needed).

Packaging

If you use a custom nuxeo assembly to package your nuxeo distribution with your plugins, you will need to modify your existing assembly.

The new nuxeo-distribution system is simpler to configure and extend than the previous one.

See here and here for more details.

Data

There are no impacting changes on the storage structure.

If you want to leverage the new default indexing configuration (separated full-text index for title and description) you will have to update your reprository configuration (or use the one provided with 5.3.1) and build the new indexes.

Configuration

The opensocial.properties file format has been changed in 5.3.1, so you may need to use the one provided in 5.3.1 rather that trying to upgrade the one used in 5.3.

Dashboard

Dasboard initialized in 5.3.0 need to be migrated to 5.3.1. You can use a simple nuxeo-shell script for that

CMIS Examples with the Nuxeo ECM platform

I recently had the opportunity to play with CMIS AtomPub bindings, in collaboration with our partners in Canada, as they were building a VBScript API to access a Nuxeo repository via the CMIS standard from a Microsoft-based environment. I had no concrete idea of what CMIS really was, but I had read a few articles about this "greatest common factor of document management concepts between all vendors," as Florent tweeted once ;-).

I don't intend, in this blog post, to give a complete description of CMIS, or its AtomPub bindings. There are good posts and tutorials for this. I want to explain a few points that were not initially straightforward to me, hoping that this will accelerate your first experience with CMIS. You should not be confronted with raw XML, as we'll see in this post, as long as you use mainstream languages: java, c#, python, etc. Indeed, there are or will be libraries to let you handle plain objects and methods. But the following approach will be very useful if you want to interface your document repository with a legacy application written in a not-so-common language that doesn't benefit from projects like Chemistry, or if you just want to understand some basics of CMIS.

Let's start with a few concepts:

• in CMIS, entities are called objects and are described in XML using the Atom syntax. You'll find in the Atom entry a full description of an object (properties, linked objects, URLs, etc.)

• each object and main collection of objects are tied to an HTTP URL. You will process information with respect to the semantic of the HTTP method used to communicate with the server:

  • GET is used to retrieve information on the object
  • PUT is used to update an object
  • POST to create
  • DELETE to delete

• CMIS server gives you, in its Atom answers, URLs to get or process all the information you need. You just have to parse the answer, looking for "rel" links that give you URLs to download attached files, children, renditions, etc. You don't have to obtain the information from anywhere else but the server, which allows you to develop highly interoperable code.

For the operations that come next, you need to deploy the Chemistry Libraries and the Nuxeo-Chemistry implementation on your Nuxeo instance (or be sure that you have any other CMIS-compliant server and adapt the lengthening of the URLs). Or, an even easier solution is to use our Nuxeo CMIS demo server which is reset everyday and frequently updated to take into account your feedback.

To send an HTTP request to the server, I will use "curl." You can also use "wget", or any other HTTP requester. In some of the requests, I need to send XML file content (most often an Atom entry). You will find up-to-date samples of those XML files in the chemistry source code; they are very instructive. I also attached to this blog post the Atom entries I used in the following samples.

Let's start with the concrete experiment :)

The initial URL, to get information on the repository will be, from a shell:

curl -uAdministrator:Administrator \
   "http://cmis.demo.nuxeo.org/nuxeo/site/cmis/repository"

It mainly gives:

• the capabilities of the repository
• the URL of the CMIS collections (types, children of the root, query)
• URL patterns that are accepted by the server, in order to get information on an object, to make a query, etc.

Here is an extract of the server's answer:

  <?xml version='1.0' encoding='UTF-8'?>
  <service xmlns="http://www.w3.org/2007/app" xmlns:cmis="http://docs.oasis-open.org/ns/cmis/core/200908/"
       xmlns:cmisra="http://docs.oasis-open.org/ns/cmis/restatom/200908/">
        <workspace>
            <title xmlns="http://www.w3.org/2005/Atom" type="text">default</title>
          <cmisra:repositoryInfo>
                <cmis:repositoryId>default</cmis:repositoryId>
              ...
                <cmis:rootFolderId>4fb1b8a1-6dfd-4da4-95b0-4ef41c27b920</cmis:rootFolderId>
                <cmis:latestChangeLogToken></cmis:latestChangeLogToken>
                <cmis:capabilities>
                <cmis:capabilityACL>none</cmis:capabilityACL>
                <cmis:capabilityAllVersionsSearchable>true</cmis:capabilityAllVersionsSearchable>
                <cmis:capabilityChanges>none</cmis:capabilityChanges>
                <cmis:capabilityContentStreamUpdatability>anytime</cmis:capabilityContentStreamUpdatability>
                <cmis:capabilityGetDescendants>false</cmis:capabilityGetDescendants>
              ...
              </cmis:capabilities>
            ...
            </cmisra:repositoryInfo>
            <collection href="http://cmis.demo.nuxeo.org/nuxeo/site/cmis/typechildren">
            <title xmlns="http://www.w3.org/2005/Atom" type="text">Types</title>
            <cmisra:collectionType>types</cmisra:collectionType>
            </collection>
            <collection href="http://cmis.demo.nuxeo.org/nuxeo/site/cmis/query">
            <title xmlns="http://www.w3.org/2005/Atom" type="text">query
                collection</title>
                <cmisra:collectionType>query</cmisra:collectionType>
                </collection>
                <collection
                href="http://cmis.demo.nuxeo.org/nuxeo/site/cmis/children/4fb1b8a1-6dfd-4da4-95b0-4ef41c27b920">
                <title xmlns="http://www.w3.org/2005/Atom" type="text">children
                collection</title>
                <cmisra:collectionType>root</cmisra:collectionType>
                </collection>
                ...
                <cmisra:uritemplate>
                <cmisra:type>objectbyid</cmisra:type>
                <cmisra:mediatype>application/atom+xml;type=entry</cmisra:mediatype>
                                        <cmisra:template>http://cmis.demo.nuxeo.org/nuxeo/site/cmis/object/{id}?filter={filter}&amp;renditionFilter={renditionFilter}&amp;includeRelationships={includeRelationships}&amp;includeAllowableActions={includeAllowableActions}&amp;includePolicyIds={includePolicyIds}&amp;includeACL={includeACL}</cmisra:template>


        ...
        </workspace>
        </service>

You need to look for the children root collection URL to start browsing following the repository hierarchy.

curl -uAdministrator:Administrator \
    "http://cmis.demo.nuxeo.org/nuxeo/site/cmis/children/4fb1b8a1-6dfd-4da4-95b0-4ef41c27b920"

Doing so, you will receive a feed of atom entries which is actually the collection of children of the root. Here is the atom entry associated to "default domain" in Nuxeo:

  <entry xmlns:cmisra="http://docs.oasis-open.org/ns/cmis/restatom/200908/">
    <id>urn:uuid:4593d0e5-fa7f-4ae1-9472-3030c270bb1e</id>
        <title type="text">default-domain</title>
        <updated>2010-01-11T11:49:20.036Z</updated>
        <author>
        <name>system</name>
        </author>
        <summary type="text">default-domain</summary>
        <link href="http://cmis.demo.nuxeo.org/nuxeo/site/cmis/repository"
        rel="service" type="application/atomsvc+xml" />
        <link
        href="http://cmis.demo.nuxeo.org/nuxeo/site/cmis/object/4593d0e5-fa7f-4ae1-9472-3030c270bb1e"
        rel="self" type="application/atom+xml; type=entry" />
        <link
        href="http://cmis.demo.nuxeo.org/nuxeo/site/cmis/object/4593d0e5-fa7f-4ae1-9472-3030c270bb1e"
        rel="edit" type="application/atom+xml; type=entry" />
        <link
        href="http://cmis.demo.nuxeo.org/nuxeo/site/cmis/object/4593d0e5-fa7f-4ae1-9472-3030c270bb1e"
        rel="alternate" type="application/atom+xml; type=entry" />
        <link href="http://cmis.demo.nuxeo.org/nuxeo/site/cmis/type/Domain"
        rel="describedby" type="application/atom+xml; type=entry" />
        <link
        href="http://cmis.demo.nuxeo.org/nuxeo/site/cmis/object/4fb1b8a1-6dfd-4da4-95b0-4ef41c27b920"
        rel="up" type="application/atom+xml; type=entry" />
        ...
        <cmisra:object>
        <cmis:properties>
            <cmis:propertyString propertyDefinitionId="cmis:path"
                localName="def:path" displayName="Path">
                <cmis:value>/default-domain</cmis:value>
                </cmis:propertyString>
                <cmis:propertyId propertyDefinitionId="cmis:objectTypeId"
                localName="def:typeid" displayName="Type ID">
                <cmis:value>Domain</cmis:value>
                </cmis:propertyId>
                <cmis:propertyString propertyDefinitionId="content_roots"
                localName="def:nx:content_roots" displayName="content_roots" />
                <cmis:propertyString propertyDefinitionId="dc:description"
                localName="def:nx:dc:description" displayName="dc:description">
                <cmis:value>Nuxeo 5 default domain</cmis:value>
                </cmis:propertyString>
                <cmis:propertyString propertyDefinitionId="cmis:lastModifiedBy"
                localName="def:lastmodifiedby" displayName="Last Modified By">
                <cmis:value>system</cmis:value>
                </cmis:propertyString>
                <cmis:propertyString propertyDefinitionId="cmis:createdBy"
                localName="def:createdby" displayName="Created By">
                <cmis:value>system</cmis:value>
                </cmis:propertyString>
                <cmis:propertyString propertyDefinitionId="dc:subjects"
                localName="def:nx:dc:subjects" displayName="dc:subjects" />
                <cmis:propertyString propertyDefinitionId="dc:title"
                localName="def:nx:dc:title" displayName="dc:title">
                <cmis:value>Default domain</cmis:value>
                </cmis:propertyString>
                <cmis:propertyString propertyDefinitionId="icon"
                localName="def:nx:icon" displayName="icon">
                <cmis:value>/icons/domain.gif</cmis:value>
                </cmis:propertyString>
                <cmis:propertyString propertyDefinitionId="cmis:name"
                localName="def:name" displayName="Name">
                <cmis:value>default-domain</cmis:value>
                </cmis:propertyString>
                <cmis:propertyString propertyDefinitionId="dc:contributors"
                localName="def:nx:dc:contributors" displayName="dc:contributors">
                <cmis:value>system</cmis:value>
                </cmis:propertyString>
                <cmis:propertyDateTime propertyDefinitionId="cmis:creationDate"
                localName="def:creationdate" displayName="Creation Date">
                <cmis:value>2010-01-11T12:49:20+01:00
                </cmis:value>
                </cmis:propertyDateTime>
                ...
            </cmis:properties>
        </cmisra:object>
    </entry>

What is this Atom entry? It is the representation of the state of the object whose ID is 4593d0e5-fa7f-4ae1-9472-3030c270bb1e, transferred by the CMIS server (on top of our Nuxeo repository). You will find mainly four parts in it:

• the Atom header, with elements id, title, updated, summary.
• the content field (when it is "document", not a "folder") or an "alternate" element, that gives a URI to a remote content
• the links associated to the document that give the URLs to request to perform operations on the object, such as service, self (the URL of the object), edit (to modify the object), alternate, described, up, down (we will see later the use of those links).
• the cmisra object description, and among them you will find the custom type property values.

For some operations (like create or update metadata), and depending on how rigorous the implementation is, you need to be sure that:

• the id element is not null
• the title element is not null
• the author or source is not null
• either alternate link or content is not null

If you miss one of those rules, your request will be rejected. Now let's go with basic operations.

1. To browse the repository, use the up and down links:

   curl -uAdministrator:Administrator \
     "http://cmis.demo.nuxeo.org/nuxeo/site/cmis/children/4593d0e5-fa7f-4ae1-9472-3030c270bb1e"
   

   In the result of the previous command, where I asked for the children of the "Default domain" node, I looked for the string "workspaces" to get the id of the workspaces object. I can then send a request to get the children of the workspaces node, etc.

2. To create a folder: Let's suppose you want to create a folder "myfolder" under the object of id "A", you will do a POST to the URL of the object that represents the collection of the children of "A". You get this URL by looking for the URL of type rel="down", and whose content-type is application/atom+xml;type=feed. Take care not to take the "tree" object, which is of type application/cmistree+xml and returns the hierarchy of descendants.

   curl -X POST -uAdministrator:Administrator \
     "http://cmis.demo.nuxeo.org/nuxeo/site/cmis/children/5af586f4-6221-4ad5-8bc6-7c34d247bc2b" \
     -H "Content-Type:application/atom+xml;type=entry" \
     -d @ $localh_path/createfolder.atomentry.xml
   

   Note the header variables. If you forget to mention them, your request will not be processed successfully. The result of the previous request is an Atom entry corresponding to the newly created object. Note also that I specified in the Atom entry the Nuxeo custom type I wanted to use: "Workspace":

   <cmisra:object>
       <cmis:properties>
         <cmis:propertyId propertyDefinitionId="cmis:objectTypeId">
           <cmis:value>Workspace</cmis:value>
         </cmis:propertyId>
       </cmis:properties>
     </cmisra:object>
   

   I can check, by browsing Nuxeo DM, that my folder has been created.

3. Document creation: To create a document with content in this folder, there are basically two methods:

   • either you create an empty document (without the binary file, just the metadata) and then you update the content using the edit-media link (two requests).
   • or you create the document with the content in the Atom entry (content element) encoded in base64 (one request only).

   Let's go with the first one. You need to prepare your Atom entry (basically get the one used for folder creation, and modify cmis:objectTypeId element with value "File"). Note that if you choose cmis:folder, or cmis:document, the Nuxeo implementation will map to Folder and File types. Also, the summary element is mapped to dc:description, title to the document name (and dc:title). The ID value you put in that case is not very important (not taken into account), but it should not be null.

   curl -X POST -uAdministrator:Administrator \
     "http://cmis.demo.nuxeo.org/nuxeo/site/cmis/children/191c4778-6f8a-4dc0-9733-6e39725366f9" \
     -H  "Content-Type:application/atom+xml;type=entry" \
     -d @$PATH/createdocument.atomentry.xml
   

   In the resulting Atom entry, look for the link edit-media. You will be able to upload a file through that URL :

   curl -X PUT -uAdministrator:Administrator \
     "http://cmis.demo.nuxeo.org/nuxeo/site/cmis/file/a0fe9d4f-1805-4c3d-b69c-eb44dd70d25f" \
     -H "Content-Type:application/pdf" \
     -H "Slug:NuxeoStudioOverview.pdf" \
     -d @$PATH/NuxeoStudioOverview.pdf
   

   Note here the Slug header parameter, to give the file name, as well as the Content-Type header parameter.

4. Metadata update: To update your document's metadata, you need to PUT an Atom entry on the edit URL of the object, adding the values of the metadata you want to update under the element cmis:properties of the Atom entry describing the object. Let's say you want to update the title and to enter the "language" metadata of the previously created document. Metadata "language" belongs to the Dublin Core schema in Nuxeo.

   Here is the most interesting part of the updatedocument.atomentry.xml Atom entry:

     <cmisra:object>
     <cmis:properties>
       <cmis:propertyId propertyDefinitionId="cmis:objectTypeId">
         <cmis:value>File</cmis:value>
       </cmis:propertyId>
       <cmis:propertyString propertyDefinitionId="dc:title">
         <cmis:value>Change the title of the document using CMIS
         </cmis:value>
       </cmis:propertyString>
       <cmis:propertyString propertyDefinitionId="dc:language">
         <cmis:value>EN</cmis:value>
       </cmis:propertyString>
       </cmis:properties>
     </cmisra:object>
   

   Note under cmis:properties you find only the desired metadata plus the cmis:objectTypeId property, which is mandatory to avoid an error (ndlr:this should be verified).

   curl -X PUT -uAdministrator:Administrator \
     "http://cmis.demo.nuxeo.org/nuxeo/site/cmis/object/a0fe9d4f-1805-4c3d-b69c-eb44dd70d25f" \
     -H "Content-Type:application/atom+xml;type=entry" \
     -d @$path/updatedocument.atomentry.xml
   

5. Document deletion:

   • To delete the document, you can use the self URL in combination with DELETE method:

     curl -X DELETE -uAdministrator:Administrator \
       "http://cmis.demo.nuxeo.org/nuxeo/site/cmis/object/a0fe9d4f-1805-4c3d-b69c-eb44dd70d25f"
     

   • To delete a non-empty folder, you need to perform the delete on the tree object, so you need to use the down link that returns an application/cmistree+xml object:

     curl -X DELETE -v -uAdministrator:Administrator \
       "http://cmis.demo.nuxeo.org/nuxeo/site/cmis/descendants/67df3f01-a4e5-4173-b685-88742cf22471"
     

Download AtomSamples

GWT Integration for Nuxeo

I've added recently a new project in nuxeo-webengine: nuxeo-webengine-gwt

This project provides the capability to develop GWT applications in Eclipse and launch an embedded Nuxeo Server as part of the GWT dev. mode server to be able to debug your Nuxeo GWT applications inside eclipse (without the need to deploy it on a real Nuxeo server).

Also it provides a mechanism to transparently deploy the compiled GWT applications in Nuxeo WebEngine (without the need to create separate WARs).

Nuxeo GWT applications are packaged as regular Nuxeo bundles.
Also you can make use of the GWT RPC mechanism in your application without worrying about RPC servlets deployment in Nuxeo.

The GWT applications can be exposed to clients either through WebEngine objects, JAX-RS objects or custom servlets.

Note that this bundle is not yet part of any Nuxeo distribution so you need to put it by hand in your nuxeo server.

You can find more documentation about this here:

http://doc.nuxeo.org/xwiki/bin/view/FAQ/GWT+Integration+for+Nuxeo

IKS Semantic Search Workshop Wrap-Up (Rome 2009)

Last week I had the pleasure to attend the second workshop organized by the IKS project in Rome. The goal of this 4 years project is to develop a software stack and a set of design guidelines to help CMS developers leverage the promises of knowledge oriented software and Linked Data.

In the following I will give a brief overview of some of the discussions that happened during those four days and a summary of the Scribo project I presented during the demo sessions the last day. A more complete coverage of the event can be found the event page of the IKS wiki.

Materialized semantic indexes

Rupert Westenthaler from the Salzburg Research team is working on a very interesting prototype to make CMS applications able to perform fast complex graph queries on a knowledge base by materializing named graph queries into flat Lucene indexes and tracking the knowledge base changes to detect when the indexes need incremental updates.

To me this sounds a lot like the permanent MapReduce views used to query the CouchDB document database. I really look forward to the release of the first prototype along with some benchmarks to compare this approach with general purpose un-materialized SPARQL engines such as Jena SDB / TDB, Sesame and Virtuoso.

Bridging CMIS and RDF/OWL

During the workshop Gokce Laleci introduced a prototype mapper from JCR to RDF, from content structure to explicit semantic knowledge. The goal is to express the underlying structure (document types and properties) specific to a given CMS content store as a standard based and interoperable knowledge view that can be directly aggregated by Linked Data crawlers.

She and her team will now work on a similar mapper for the CMIS protocol using Nuxeo DM and Apache Chemistry as primary integration platform. Another interesting lead would be to translate SPARQL queries in CMISQL when the mapping makes sense and hence allow any CMIS content repository to behave as a SPARQL endpoint.

In the long term, I am not sure whether we want to keep the content and knowledge in separate stores as we do currently in Nuxeo (Nuxeo Core and Jena). It might be simpler and more efficient to combine them both in the Core and use such a configurable knowledge mappers along with materialized graph queries to implement the semantic features of Nuxeo.

Ontology-free semantic indexing

Stephane Gamard (@sgamard) introduced the services offered by the SalsaDev platform. Their startup focuses on leveraging an algorithm able to semantically index any text document such as blog posts, web page snippets, wikipedia articles and look up semantically related documents in all indexed content without relying on explicit ontologies or topic classification. Their approach offers the same advantages as Latent Semantic Analysis but is also scalable to very large document collections while LSA suffers from quadratic lookup times that makes it unusable in practice.

This approach is very similar to a semantic hashing prototype I have been working on my idle weekends for quite some time now. The short term goal is to implement an image search by similarity feature for the future Nuxeo Digital Asset Management product. On a longer term the same algorithm should be adapted to also work for text document similarity search.

Those purely data-driven approaches are interesting for at least two reasons:

• they allow for a natural implementation of the unstructured "query by example" paradigm,
• they can be combined with more structured semantic extractions to perform disambiguation in a named entities recognition component for instance.

Using UIMA for economic intelligence

Tommaso Teofili (@tommasoteofili) from the Apache UIMA team demoed a real application of semantic knowledge extraction to monitor the temporal evolution real estate market prices in the Rome area. The assets prices data categorized by surface and number of rooms is automatically extracted from the raw unstructured content of public ads web pages and aggregated in a relational database that feeds a charting and reporting user interface.

The data extraction magic is performed by a UIMA chain that wraps the online semantic engines provided by the AlchemyAPI web service. Such semantic lifting services are typically what Nuxeo aims to provide as part of the platform without relying on third party service providers.

Incidentally Peter Mika from Yahoo! Research is working on a similar prototype to find his next flat in Barcelona.

Nuxeo and automated Semantic Knowledge extraction

As part of the demo session, I chose to present some of the ongoing work done by Nuxeo and its partners as part of the Scribo project. One of the goals of this project is to extract the occurrences of entities (such as persons, organizations and places) semantic assertions between those entities ("Person A" is the CEO of "Company B" or "Person B" has declared that "he will reform the Health care system"). To that hand we chose to package annotators as chained UIMA Analysis Engines and store the extracted semantic annotations as RDF assertions using the classes of the DBPedia ontology. Here are the slides introducing the context of the demo:

Nuxeo Iks 2009 11 13

View more documents from ogrisel.

The demo itself is two-fold. The first part features the Scribo Workbench mainly developed by XWiki to configure and test a chain of UIMA annotators to extract semantic knowledge from the text content of documents coming from heterogeneous content repositories such as a filesystem folder, a CMIS repository (Nuxeo DM) or an XWiki server accessed through its RESTful API.

The user can then combine such a document source with one or several registered annotators into a UIMA chain (a.k.a Collection Processing Engine), run the process and view the results as annotated text document directly in the Eclipse UI. The user can also validate or invalidate the extracted annotations and hence incrementally build a validated knowledge base of semantic statements out of his unstructured content. The following screencast shows the details of this scenario using the Stanford Named Entity Recognition annotator on 2 wikinews articles:

The second part of the demo showcases the deployment of the previous UIMA chain directly inside a Nuxeo DM 5.3 instance. PDF documents are directly semantically annotated at import time thanks to an asynchronous event listener that calls a new UIMARunnerService packaged as an OSGi components deployed by the Nuxeo Runtime.

The extracted named entities are stored in the default Nuxeo Jena store. Some work is still needed to make the annotations show up correctly in the "preview tab" and make it possible to validate / invalidate extractions from the "knowledge base" tab.

Remember this is just the beginning and we plan to support all languages significantly represented in Wikipedia along with finer grained entity classes. You can also get an overview on the global semantic R&D effort at Nuxeo on our Jira.

Last but not least, the showcased demo is deployable on your own Nuxeo DM 5.3 instance by deploying a simple plugin as explained on the UIMA page of the Nuxeo wiki. Beware that this is really alpha alpha work and should not be deployed on a production setup.

Nuxeo DM 5.3 release notes

Nuxeo EP and DM 5.3 have recently been announced. Here are some additional technical notes about this release.

If you can't wait, go download the packages now.

Highlights

Standards for Social Business: Support for OpenSocial allows the creation of gadgets to build web mashups within enterprise applications. Nuxeo DM 5.3 can serve as both publisher of gadgets as well as be an OpenSocial container, allowing the Nuxeo DM repository to host and participate in enterprise mashups.

Windows Sharepoint Services support: Access Nuxeo DM 5.3 via Microsoft Sharepoint for basic library services. Native integration with Windows Explorer ensures information workers can use familiar browsing habits to access Nuxeo DM content and perform common file operations. Microsoft Office integration allows files to be opened or saved directly to/from Nuxeo DM and lets users see information about their content directly from the Document Panel in Microsoft Office.

Federated publishing: Centrally control and publish content to remote Nuxeo applications, file systems, HTTP servers, web portals, and more thanks to a pluggable and unified Publishing Service. It is suited to deployments with a distributed information architecture, balancing ease of publication with requirements to for content control.

Tagging Services: Enhanced metadata capabilities ensures information workers can categorize their content in ways that make sense for average users. Tagging recommendations and tag cloud support enrich an organization's ability to describe and retrieve their information assets.

New Performance Record for Large Deployments: Faster search and retrieval benchmarks for Nuxeo DM repositories holding several million items for hundreds of concurrent users using inexpensive commodity hardware specifications.

Email Capture Enhancements for MS Exchange and Gmail: Improved support for email folders. Fetch email from Microsoft Exchange or Gmail to/from the Nuxeo DM 5.3 email archive folders to ensure contextual management of business communication.

CMIS Support: Ensure true interoperability across multiple ECM, ERP and search systems, Nuxeo DM 5.3 offers a CMIS Server via an add-on component, based on the CMIS draft 0.62.

Search/Indexing Improvements: Granular metadata search and indexing allows users to find and retrieve content based on one or more fields.

Improved Developer Productivity: Several enhancements have been made to make the experience of developing on top of the Nuxeo platform more productive and enjoyable.

From 5.2.1 to 5.3

Our initial plan was to release a 5.2.1 at the beginning of the summer.

But, for good or for bad, we were very busy working on customer projects and missed the windows of opportunity to make a release at the time.

So when we finally could focus on making a new public release of Nuxeo DM, we realized that we had done too much work to just call it a simple maintenance release.

That's why this present release is numbered 5.3.

In parallel, critical fixes have been back-ported in 5.2 as a service pack (Nuxeo 5.2 SP1).

Full backward compatibility

This release introduces new services and APIs, but we did not break any existing API.

This means that developments done against 5.2 should run without major problems against a 5.3.

Unlike the 5.1->5.2 migration that required compatibility packages, the 5.2->5.3 migration should be completely pain-less (see below).

Additional features in Nuxeo DM

As always, we tried to include in Nuxeo DM as many useful features as possible.

Nevertheless, all the new features provided by Nuxeo EP are not directly visible in Nuxeo DM 5.3.

There are several reasons for this:

• the Nuxeo DM distribution is already pretty big

• some of the technical infrastructure improvements can not be simply demonstrated

This means that new features in Nuxeo EP:

• have been included by default in DM when it makes sense

• are available as add-ons in the other cases

For the features released as add-ons, all necessary new APIs in Nuxeo DM are already part of 5.3.

What's new in 5.3?

Web features

During the last months we've worked with our partners and customers on several intranet and portal projects.

This means a lot of small improvements have been made to WebEngine and the Theme engine.

Tag Service

You can now add tags to Nuxeo Documents and:

• search documents based on their tags

• navigate the document database via a tag cloud

The tag service is only available when using VCS storage.

In the middle term, it could become a feature of Nuxeo Core.

Here is a demo:

Blogs

We finally released a simple Blog implementation using WebEngine framework.

This blog is a simple example of how Documents can be managed in the back-office (in the default JSF WebApp) and accessed via a public Web interface.

WebWidgets

The Theme engine of 5.3 comes with support for WebWidgets.

It allows to include UWA JavaScript widgets containers in the Themes.

WebWidgets are very close to OpenSocial gadgets (see below). In the future we will make converge these 2 javascript portlets models.

Because OpenSocial and WebWidgets are providing very similar features, WebWidgets are not part of the default packaging.

OpenSocial

Contributed by the community

If you were at last year's Nuxeo DevDay conference, you probably know that Damien Metzler [video proof] and his team from Leroy Merlin have been working on Nuxeo WebEngine and OpenSocial for some months now. You can find more informations about their work on [Damien's blog].

They contributed a lot of their work on the integration of OpenSocial in Nuxeo:

• Apache Shindig (OpenSocial server) as a Nuxeo Service

• GWT based OpenSocial gadget container

• WebEngine based OpenSocial portal

• Integration of Shindig with Nuxeo's authentication and user management

• Gadget persistence API based on Nuxeo's DocumentManager

Some slides about Nuxeo and OpenSocial:

Nuxeo 5.3 integration

Based on the work contributed by Damien's team, we rebuilt the Dashboard using the GWT gadget container and Nuxeo's REST API.

Basically, all most previously available portlets are still here but now:

• users can customize their dashboard

  • add/remove new portlets (gadgets)
  • change layout

• it's very easy to add specific gadgets (HTML + JavaScript, instead of JSF + Nuxeo Themes)

Here is a video showing their work in action:

Currently, OpenSocial integration into DM is limited to Gadgets. It does not expose all the infrastructure work already done, nor does it exposes all advantages we can gain from the OpenSocial standard.

In the next months, we expect to also take advantage of OpenSocial social features.

Multi-instance management

Remote Publishing

The publishing service has been replaced by a completely new Publisher Service.

The new API lets you publish a Nuxeo Document to an abstract tree.

This tree could be:

• a local Nuxeo Sections tree (as before)

• a Section tree hosted on a remote Nuxeo server

• a filesystem tree

• a custom tree pointing to an external application

This service was designed to support decentralized publishing in Nuxeo, but can also be used to publish Nuxeo document to a existing portal or web site.

Nuxeo Replication

This new add-on provides an application-level replication service.

It is used to replicate changes (Documents, Audit, Directories...) from one Nuxeo instance to another.

Because this replication is managed at the application level, you can decide which part of your data you want to replicate. This replication scope is typically defined by a NXQL request.

This service has already been used to:

• provide an offline client based on Jetty DM bundle (replicate only the documents accessible by the user).

• provide staging between several Nuxeo instances (push a whole tree to staging).

Windows integration

Browser and Office helpers

Since we are not .NET and MSOffice experts, LiveEdit and MSIE plugin have always been a pain to package.

The good news is that we found someone to help us with this, and as a first step, we did a big code cleanup, some bug fixes and all .Net builds were now integrated in the CI chain.

As a consequence, it will be easier to maintain and improve these plugins.

Any feedback on these plugins will be helpful: we don't have many MSIE and MSOffice users on site.

WSS Extensions

Windows SharePoint Services (WSS) is a set of protocols published by Microsoft that describe how SharePoint communicates with the rest of the Microsoft world.

WSS has a broad scope and contains different technologies (FrontPage extensions, WebDAV, CAML, WebServices...).

For implementing WSS extensions for Nuxeo, we focussed of the interfaces exposed by SharePoint to client applications like MS-Office and Windows Explorer.

The goal is to let MS-Office and Explorer talk to a Nuxeo server as if it was a SharePoint server.

Since part of the work is boring protocol implementation (like implementing Front Page extensions), the Nuxeo WSS extensions are implemented by two separated module:

• a generic handler that does not rely on Nuxeo framework code and provides a SPI (Service Provider Interface)

• the Nuxeo WSS backend which implements the SPI on top of the Nuxeo EP services (Repository, Relations, Workflow, UserManager)

In order to use WSS you will need MS-Office 2003 or 2007.

See the screencast below for these features in action:

VCS improvements

Nuxeo 5.2 was the first version to ship with VCS, or "Visible Content Store", our SQL-based backend for Nuxeo.

Nuxeo 5.3 comes with a lot of improvements of VCS.

Performances

We did a lot of performances testing on VCS, using FunkLoad (our open source functional and load testing toolkit) and the importer (see below).

Based on our results and on the feedbacks from support, we did some performance improvements.

Security checks: low-level security checks have been optimized so that even filtering several millions of documents can be very quick.

Path-based queries: VCS now manages a new "ancestors" table that allows quick queries on the path.

Proxies optimizations: proxies search now avoids costly joins.

Thanks to these optimizations, browsing and searching on a repository with several millions of documents is not an issue, even with a cheap server.

Indexing

In addition of the CMIS Join support (see below), VCS now supports multiple fulltext indexes.

Database support

VCS has now been tested "in real life" with several database vendors.

We fixed a lot of small issues related to specific vendors.

Nuxeo DM is now completely CI tested (unit tests and functional testing) against target DBs.

New APIs

Some new APIs have been introduced to manage import and replication.

Nuxeo DocumentManager now support CMIS queries (including JOINS).

CMIS

We are actively working on Chemistry and CMIS, the public review of CMIS 1.0 has just started.

We will publish a demo server based on Chemistry + Nuxeo in a few weeks.

Please see this wiki page for more informations.

Follow the blog entries tagged "CMIS" on the Nuxeo blogs.

Tools

Importer framework

A lot of projects using Nuxeo need at some point to inject documents into a Nuxeo repository:

• initial data migration (from a filesystem or a legacy document management application)

• fetch and inject mail, feeds, records...

• load testing

The importer add-on provides the infrastructure for doing such tasks easily and efficiently:

• thread pool for multi-threaded import

• pluggable threading policy

• pluggable DocumentModel factory

• pluggable crawler

• configurable batch transaction management

• bulk mode management for events and associated handlers

• JAX-RS bindings

It includes ready to use implementation for:

• a simple file system importer

• a file system + meta-data importer

• a random text injector based on dictionaries

This package is available in mercurial.

JCR2VCS migration tool

This tool provides a way to migrate JCR based repositories to VCS.

This add-ons contains 2 parts:

• one exporter (JCR on 5.1.6 and 5.2)

• one importer (VCS on 5.3)

Documentation about data migration is available here.

Improving the developer experience

Jetty / Tomcat support for DM

Nuxeo distribution now supports deploying Nuxeo DM on Jetty and Tomcat.

These distributions does not include any Java EE supports (Transactions, EJB3, JCA...), but provide the same ECM features.

These packagings are mainly targeted to development environment since startup is significantly faster than a complete JEE container (less than 40s for full startup).

In the near future (one month), we will add a Transaction Manager to the tomcat package, this will make the tomcat package completly ready for production.

JBoss speed-up

The JBoss deployer has been optimized to start faster. A full Nuxeo DM startup on a laptop with JBoss takes now 1m45 instead of 2m30.

NB: this speed improvement should be very significant on windows boxes having an virus scanner intercepting all file system accesses.

Seam Hot reload support

Seam components hot reload is now supported.

This can significantly improve development speed when using the JSF framework.

See: http://doc.nuxeo.org/xwiki/bin/view/FAQ/SeamJSFHotReload for more info.

GWT

GWT integration in Nuxeo has been improved so that you can now run GWT Application in hosted most with a bundled Nuxeo server deployed inside Eclipse as a Jetty server.

It is now easier to build a GWT app that uses Nuxeo Platform:

• in development mode, eveything is integrated into Eclipse

• we provide a single API jar for all Nuxeo services

Documentation

We improved the available documentation:

• Single entry point for learning Nuxeo www.nuxeo.org

• "Getting started" entry point

• Documents about DB tuning and settings

• Documents about data migration

Translations

Translations files have been externalized so that it will be easy for the people interested in submitting translations to do so.

Your turn now!

Trying it out

To try it out, grab the packages:

• http://www.nuxeo.com/en/products/dm/try-it

Joining the community

• Read the doc.

• Get the source code.

• Join the conversation: mailing list, forum.