Reinout van Rees is just doing an amazing job in his blog for EP feedback http://vanrees.org/weblog/

I have been at Michael Salib's talk about Xapian, "Stupidity and laser cat toys: Indexing the US Patent Database with Xapian and Twisted"

Xapian is a probabilistic text search engine.

Michael used to index the US Patent Database, wich is pretty big indeed.He wrote a python wrapper called Xapwrap, that you can get here:

http://divmod.org/projects/xapwrap

Michael explained that Xapian was prefered to Lucene because It easier to wrap into Python and provided faster queries and a better precision.

I'm waiting for Michael to upload the slides on the EP sites to give more precise feedback on this.

More info on PyLucene here: http://www.sauria.com/~twl/conferences/pycon2005/20050325/Pulling Java Lucene into Python.html(PyCon05 notes)

feature-wise, Xapian has eveything needed to run a scalabale text engine.(stemming based on snowball, meta-indexes, etc..) It optionnally uses twisted's python.log for logging.

• Lucene features: http://lucene.apache.org/java/docs/features.html
• Xapian features: http://www.xapian.org/features.php

I have the feeling that Xapian would fit pretty well as an external indexer for z3

I have uploaded my slides from yesterday's EP2005 presentation, you can grab it here:

http://www.cps-project.org/sections/news/ep_2005_cpsmailaccess
It presents the webmail system framework and how it scales for wide intranets, using stuff like a stateful connector pool.

I'm writing a little blog entry atthe end of conference day 1.
A lot of things hapenned already. I could not see all the conference I wanted of course.

About us:

Stéfane presented CPS on a one hour track, trying to show all cool aspect of CPS. Julien tallked on a 30 minutes track about workflows and Florent about his thaught on the future relation/repository tool in Zope 3, based on Zemantic. Slides are available now.

For myself I did a presentation about CPSMailAccess, the CPS webmail. I was pretty happy about the audience reaction, as they found it very usable and ergonomic. Some zope developers told me they will going to try it in other CMS environments (Plone, etc..) ...cool :)

The work we did wih Godefroid during the Sprint on "azax" will surely be reused later in the webmail.

At the end of the day, we went to Guido's keynote. He spoke about some PEPs and the Python Foundation.

Now we're are at Chalmer's hangout with Python Community people and discussion are very interesting.

Yesterday has started Zope 3 sprint in Chalmers before Europython.

Among listed subjects people were interested into:

+ repository
+ tools for document type creation, schemas and layout
+ Xicken in Five
+ Workflow
+ Relations
+ Implement a macro portlet
+ Creating an AJAX infrastructure
+ XForms

The day was a dicussion on all this subjects and some demos. Among other demos, Paul Everitt showed us what he has done using AJAX to create a page based on XForms that interacts with preloaded client side datas.

I am highly interested in this subject but I do need to learn more about XForms before I can help on the topic.

After that Godefroid Chapelle and I discussed how could we bring  AJAX into Zope, like Ruby on Rails did, to let the developers use its principles without having to write javascripts.

We have started to list use cases of interest and work on a model. CPSMailAccess has a lot of use cases that were implemented in "raw" javascript and will probably be a good field of investigation for testing what we are going to do.

Since more people will join us today and tomorrow, and since most of the people have not yet decided what they were going to do, I guess i will post tomorrow a more precise list of points with more infos. (the list I gave is most likely to be the draft list)

Background

CPSMailAccess let the user configure the SMTP server he or she wants to use. It can be the same SMTP server for all users, or a different one for each user. Having multiple configuration can sound overkill, but since the application let you browse any mailbox, you might need to associate a different SMTP server to each MailBox

What Zope 3 provides

Zope 3 introduces a pretty cool mechanism: a delivery queue that writes mails into a filesystem dir (a MailDir in fact) and let a thread send them asynchronously.

This is rock-solid, since :

• the work can restart even if Zope hangs: no send will be lost
• the user does not wait

that's exactly what's needeed for large scale intranets. The pattern used is:

• MailWriter: writes mails into the folder
• MailReader: thread, sends mails found into the folder
• MailSender: interface,provide a send method that calls MailWriter, and creates the MailDir

Special headers are inserted in the mail, so extra informations can be shared between the MailWriter and the MailSender.

What CPSMailAccess needs

The only missing feature for CPSMailAccess is to add the SMTP host and port informations into mails that are written into the MailDir.

Those are read by the sender and removed before the mail is sent.

The beta 2 version will include this feature, with a MailWriter-MailReader-MailSender pattern derived from Zope 3 classes.

And thanks to Zope 3 pythonic style, this module can be used in any Python application.

It's been two days that i am googling to try to find an IMAP Proxy program that would let me speed up this kind of search:

find me all mails that contains the word "cps" in their body.

in my 30k mails mailbox, it lasts over to 30 seconds, depending on server charge.

Why mail bodies are not indexed on Zope side

I have decided a while ago to stop indexing mail bodies on zope side, as this is not reliable for big intranets, big account, etc... It is not a problem of speedness, as a primary indexing of a big mailbox was taking ages, because zasync could be use there, but it's more a problem of making the ZODB growing very, very fast and big.

A filesystem storage linked to the ZODB ?

This would bring a lot of hard work on setting up everything to work fine together. (one more player, you need a chaire and more room around he table)

What could be done on IMAP side: a Python Proxy

Back to my pure IMAP search: I find this too slow, and I don't really understand why they are no options in IMAP servers to index the messages to speed it up things.... I can't find anything related on such tools like Courier IMAP.

I don't want to use ZOE either, I want a server side tool that can work with all mail clients.

I tried to find some proxies that would do this kind of work, to be placed between client requests and the real IMAP server, but I have just found some IMAP Proxy that would make statefull connections for webmail apps that reconnects everytime something is done. (Connection are statefull in CPSMailAccess)

The code of such a tool is quite simple and short but has to be very robust:

A SocketServer.TCPServer that would listen to IMAP ports (both SSL and not SSL) can:

• Catch all body search queries and search in a system file that contains index of message bodies instead of calling the real IMAP server
• Index incoming message before passing them to the IMAP server.

This would speed up all full text searches and bring them as fast as classical header searches, even if it's coded in Python.

Now, this sounds easy said this way, but this is a first thaught so I guess I do not see all problems involved.

I'm happy to announce the release of CPSMailAccess 1.0 beta 1

the next release will introduce very interesting features (roadmap)

the annouce:

June, 8 2005 -- Nuxeo is happy to release CPSMailAccess 1.0 beta 1.

CPSMailAccess is a full featured open source webmail application for CPS3 under the GPL licence. It provides a flexible web interface to browse and send mails, fully integrated to CPS portals, as it can be used by any allowed member of the portal.

CPSMailAccess provides among others features:

• AJAX based mail editor
• drag'n'drop for mails and folders
• incoming mails filtering
• private and public address books
• etc.

CPSMailAccess is built on Five, thus making the Product easily portable to Zope 3. Its architecture also allows to integrate it to any Zope-based platform as its dependencies to CPS3 are close to none and thought to be easily overriden.

For more functional and technical information, see the CPSMailAccess web site:

• http://www.cps-project.org/sections/projects/cpsmailaccess

CPSMailAccess can be downloaded here:

• http://www.cps-project.org/static/misc/CPSMailAccess-1.0.0-b1.tgz
  

You can also get the latest changes on the svn repository, browsable here:

• http://svn.nuxeo.org/trac/pub/browser/CPSMailAccess/
  

And get the cuting edge version from the trunk with this command:

svn co http://svn.nuxeo.org/pub/CPSMailAccess CPSMailAccess

Look at the INSTALLATION.txt file in the package for more information on how to install CPSMailAccess, or on the web site:

• http://www.cps-project.org/sections/projects/cpsmailaccess/install
• 
  

If you'd like to help with CPSMailAccess development or porting to other plateforms, please join the cps-devel mailing list at:

• http://lists.nuxeo.com/mailman/listinfo/cps-devel

(a dedicated list will be created if needed in the future)

For any user feedback or information, use the CPS users mailing lists:

• http://lists.nuxeo.com/mailman/listinfo/cps-users: english speaking list
• http://lists.nuxeo.com/mailman/listinfo/cps-users-fr : french speaking list