Lennart Regebro

Triage of bugs in an open-source world.

The word "triage" in programming has been around for a while, but seem to have gained popularity lately. What it is, is basically a new word for "bugprioritizing", being easier to say and spell. But if we look at what the word comes from, we see that behind this lies a concept that is often forgotten in the usage.

"Triage" comes from french and means sorting, so nothing new there. Howerver, it comes to the programming world from the world of emergency health care, where it is used to allocate health care resources in emergencies, when there are more people needing care than there are health care available. In computers then, it becomes a system of allocating programming time to bugs, when there are not enough programmers to fix all the bugs.

How should this be done? Well, first of all, we of course need to take a look at how serious the bug is. What categories you have depend largely on what bugtracking system you use, and it's also a matter of personal taste. Personally, I would prefer these categories:

A blocking/critical bug is a bug that means the system or part of the system gets unusable. It has no workarounds, and affects most users of the system.
A serious/medium bug is a bug that is a big problem for many users and has no workaround, or affects everybody, but has a workaround.
A minor bug is a bug that affects few users and has a workaround, or has no workaround but only is a problem in very extreme edge usercases.
A trivial bug is not an actual problem for anybody. Could be spelling errors or ugly design.

Bugs also usually have a priority field. Prioritizing or triaging a bug is usually just setting this field. But triage is not prioritizing, it's resource allocation. And although in medicine, the resources are helath care workers, and the criticality is time, in programming, the resurce is programming time, and the ciriticality is rather the knowledge to fix the really difficult bugs.

That's OK as long as you work in a company, because there you can take the bugs allocated to you, in the order your boss prioritized them. If you get problems, you ask your co-workers. But in the open source world there is bigger differences. Also in the open source world, people assign bugs to themselves, and if you think a bug is over your head, you will simply never try to fix it.

Therefore, the bugs importance should not be the only data you tag the bug with, but also how hard the bug is to fix. Most bugtrackers only have severity and priority. Some, like JIRA, has a field where you can estimate it. Although that's more of a XP project management features than a help in triage, it can be used as such. If you have an hour over to bugfix, there is no point in trying to understand a bug who is estimated to take hours to fix.

But what would be even more helpful in an open source situation would be a difficulty field. The experienced bugfixers can look through the bugs, note severity and difficuly, and then not fix the easy bugs. It's tempting to fix the easy ones because quick fixes are satisfying and you feel productive. But if the wanna-be contributor can fix bugs marked as "easy" and understand and fix them, then we have not only gotten a new contributor, we have saved time for the experts.

Finding the last changed object in a ZODB.

Today I had to figure out why something that should not create a write transaction did create one.

The first step is easy, you pack the database, do the thing that should not modify anything and look in the "undo" tab of your Zope site. There you'll see if something was written or not, it comes up as an undoable transaction.
Then it is a matter of figuring out what happens. In this case, it was quite a complex script with many parts. I wanted to see if I could narrow down the problem by seeing what was actually changed. So, from my lib/python directory, I ran this script:

from ZODB.FileStorage import FileStorage
storage = FileStorage('/path/to/var/Data.fs', read_only=1)
iter = storage.iterator()
try:
    while True:
        transaction = iter.next()
except:
    pass
for rec in transaction:
    print repr(rec.oid)

This prints out the "oids" of all modified objects in the transaction.
Then I can run zope in debug mode:

bin/zopectl debug

When I get the python prompt, I can get the objects that were modified:

app._p_jar['the oid pasted from the previous script']

This allowed me to narrow down the search for the culprit quite a lot. It still remains to figure out where in the code the modification is, but at least with the above you can get a rough idea of what is happening. See also Debugging ZODB bloat which documents roughly this process.

Easter python eggs

It's easter, and only fitting that I play with eggs, so today I have been looking at making python eggs. Eggs are a new way of distributing python software that gets more and more popular. This is mainly because installing it is simple, and because an egg can contain a list of requirements, and the installer will check if these requirements are installed, and if not, it will look them up in the standard Python module library (known as the Cheese Shop), and download and install them.

This means the end of long hours of downloading, compiling and installing all those modules you need to get your python software working. You download just the module you want as an egg and install it, and all requirements will be installed with it, if they have been eggyfied too.

Making eggs is unfortunately a bit tricky. This is because there is no good examples of how to do it, so you have to read through all the documentation, and experiment a lot. This is a shame, because in the end, it turns out to be very easy.

All you need is to install the python module setuptools, and create a a setup.py for your package. Creating it can be slightly tricky, if your package layout is complicated. Also, there is not always clear what the parameters in it should be, and there is quite a lot of searching and trying out (and for my finally some honest debugging of setupttools too figure out what was going on, really).

The parameters are for the most part the same as for making a package with distutils, so if you have a distutils package, all you need to do is to switch out the "from distutils.core import setup" to "from setuptools import setup" and you are done.

Here is an an example of a full setup.py, for the CalCore module:

#!/usr/bin/env python

from setuptools import setup

setup(name='calcore',
      package_dir={'': 'src'},
      packages=['calcore'],
      version='2.0.1',
      install_requires=['zope.interfaces >= 3.0',
                        'zope.schema >= 3.0',
                        'zope.i18nmessageid >= 3.0'],
      dependency_links=['http://download.zope.org/distribution/',],

      # metadata for upload to PyPI
      author='Martijn Faassen, Infrae;Lennart Regebro, Nuxeo',
      author_email='lregebro@nuxeo.com',
      description='Calendaring system',
      license='GPL2',
      keywords='calendar icalendar',
      url='http://www.cps-project.org/sections/projects/calendar_server ',
      long_description="""CalCore is an advanced, flexible calendaring component for Python. It
        allows the Python developer do write advanced calendaring applications
        either using their own event storage or integrating with external
        calendar servers.
        
        Features of the CalCore include among others:
        
        * Support for making private calendars, shared calendars, resource
        booking and more.
        * integration with iCalendar clients (Apple iCal, Mozilla Sunbird,
        KOrganizer...) using the iCalendar protocol,
        * invitation workflow,
        * meeting support, including helper functions to look for free time,
        * recurring event support (thanks to SchoolTools recurrence
        implementation, http://www.schooltool.org),
        * etc.
        
        CalCore is being used as the core of Nuxeos CalZope and
        CPSSharedCalendar products, products for integrating with Zope and CPS.
        These products provide a complete web-based user interface to the
        CalCore calendaring.""",
      classifiers=['Development Status :: 5 - Production/Stable',
                   'Intended Audience :: Developers',
                   'License :: OSI Approved :: GNU General Public License (GPL)',
                   'Operating System :: OS Independent',
                   'Topic :: Office/Business :: Groupware'],
      platforms='All',
      )

You'll note the install_requires parameter, that list the requirements. You can also use ==, of course, and you can list requirements that are only used for testing, and lots of other stuff. Also note the dependency_links parameter, which enables downloads from other places than the Cheeze Shop, in this case Zope.orgs repository of Zope3 eggs.

Version 2 of the calendar modules released!

Friday and today was finally the days when an official release of the calendar was done after the refactorings I have been doing since december. There are some big internal changes:

New i18n support

The translation is now done with the Zope3/Five i18n support that comes with Zope 2.9. This means that it no longer requires Localizer, which makes it much easier to use the calendar outside of CPS. In fact, you now have a full i18n support in pure Zope!

Using local utilities instead of portal tools

Also in Zope 2.9 is the support for local utilities. This is the Zope3 replacement for portal tools. This means that you no longer depend on the name of the utility, but you depend on what interface it provides. This too makes it neater to use the calendar outside of the CMF framework. Most of all this provides cleaner code internally, and is another step towards having the same code for Zope2 and Zope3 (although there is still some way to go).

Improved installation support.

CPSSharedCalendar now uses GenericSetup to do installations and upgrades. GenericSetup is still somewhat immature, but it already is a great improvement on the programmatic installers we used before.

CalZope doesn't use GenericSetup yet, amongst other things becuase there is no generic support for installing local utilities, and the support I wrote in CPSSharedCalendar is temporary pending a refactoring of local utilities that Philikon and J1m is working on. However, CalZope does now provide an install script, so installation is easier. CalZope is only a base for you to integrate into your Zope site (as we have done with CSSharedCalendar) but this installation script will at least kick-start you. ;)

Calendar publishing

In CPS you can now publish calendars into sections, thereby having public access to reading workspace calendars.

Packaging

There has been some chaneg in the packaging. Officially there are now three bundles:
CalCore-bunde, for use outside Zope, CalZope-bundle for use in Zope2 but without CPS, and CPSGroupware for use in CPS. The CPSGroupware bundle includes all you need for the calendar and the equally Five-based CPSMailAccess, our new webmail client.

You can find all of them here: http://www.cps-project.org/static/misc/

The future

The calendar modules continue to live on the bleeding edge of Zope2/Zope3 integration, and I hope to have the time to continue to keep it there right up to the time where we can use the same code to run under both Zope2 and Zope3. It's getting nearer.

I also hope to work on a CMF/Plone version. We'll see more on that soon.

Using a special ZODB with ZopeTestCase

Sometimes you want to do tests that involve a special ZODB. The typical example of this is when you do tests for upgrades. As you can't create an old ZODB programatically without having the old products at hand, the most obvious choice is to use an old ZODB, and try the upgrade on that one. Importing a ZEXP of an old database may be an alternative, but actually using an old ZODB is closer to the real thing.

So, how do you specify which ZODB you want to use? Well, the standard way is to place a file called custom_zodb.py to provide your custom storage, but this turns out to be very tricky, as ZopeTestCase will try to provide it's own custom_zodb.py. A careful tapdance is needed, centered around the configuration variable 'testinghome'.

First in your test, import the Testing module. The testing module will, when you import it, set the testinghome variable to $SOFTWARE_HOME/lib/python/Testing:

import Testing

After that, set testinghome to wherever *you* want it, in this case a subdirectory called "db" beneath the location of the test module.

import App.config
cfg = App.config.getConfiguration()
cfg.testinghome = os.path.join(os.path.dirname(__file__), 'db')

During the import of the ZopeLite module, the Zope Application will be started, and it will now use our testinghome, find our custom_zodb.py and use our custom ZODB. So we import ZopeLite, or rather, we import ZopeTestCase, which in turn imports ZopeLite, which in turns starts Zope.

from Testing import ZopeTestCase

After this, you can use the ZopeTestCase as usual:

class MyTest(ZopeTestCase):
    ....

User interface of the day

I want this user interface on a OS with orthogonal persistance and zope style aspect oriented component architecture. No further comments needed, really.

http://pouet.fokkus.net/multitouchreel.mpg

Guido and Webframeworks

Guido van Rossum has started to look for web frameworks. Now, he doesn't seem to enthusiastic about using Zope. And at first it looks like he is making the right choice, because his stated requirement list is too small:

Independence from web server technology.
Templating with reuse.
Cookie handling.
Query parsing.
URL dispatch.

Note especially the lack of any kind of data storage, authentication/access control, forms and session handling. Well, with these requirements, he shouldn't use Zope, that's for sure. But does he really not want any storage, or access control?

Wel....in his second post he extends the requirements quite lot, and suddenly it seems like persistence, acces control and session is in the game again:

"Maybe we need more standardization efforts like WSGI, that let you plug in different animals, or roll your own, for various pieces of useful web functionality: for example URL dispatch, templates, persistence, authentication, sessions, forms, style sheets, i18n, and client-side scripting (AJAX or not)."

Right, for a really good, generic web framework, we surely do need all this. And Zope 3 does it. But Guidos comment about Zope is: "From this perspective, Zope and Twisted are off the scale: they support the mix-and-match approach, offering several alternative solutions for many of the important issues (templating, persistence, authentication, etc.). But they only work if you drink lavish quantities of their particular flavor of kool-aid, and that's not good enough for me. I don't want to depend on any particular flavor of interfaces, adaptation, serialization, discovery, etc."

Yes. They do require that you use their "particular flavor of interfaces" and so on. But how are you to do it otherwise? If you want the persistance to be pluggable, then everything that uses the persistance has to conform to the same interface. If you want to have pluggable URL dispatch, then everything that uses this URL dispatch must conform to the same interface. If you want to have pluggable authentication, then all the plugins must be written for the same plugin system.

Because that's what standardization *is*, conforming to a standard.

What Zope 3 does is not only to standardize all the mentioned things, but to standardize the way to make these standards, that is by defining interfaces, making objects and adapters, and connecting it all together in ZCML. Now, if you don't like it, thats fine, but any effort to do the same things will necessarily need to create a framwork of similar complexity. It might be possible to show that Zope 3s approach of how to do it is wrong, but Zope 3 does do it.

I suspect there are two issues here:

Zope is often seen as being unpythonic. Zope 2 certainly was, but Zope 3 is trying really hard, and in my opinion succeeding, in being pythonic. There is no more PYTHONPATH magic, there are no more half-assed "pythonscripts".
In Zope 3, you create pure Python products, and you the create a user interface for these by creating "views", which will typically be a page template. That's really the whole gist of it. Now, if you want to do override some default behavious, like the way URL handling is done, then yes, you will have to write the necessary adapters for doing that according to the Zope 3 standard of doing it. But isn't that unavoidable?
Guido does not like Zope Page Templates, and I can understand why. But, as noted, the templating system in Zope 3 is pluggable, and you can for example use ClearSilver thanks to Martijn Faassens Clarity product.

I'm all for reinventing the wheel if the new wheel gets better. But really, all you Python people who shun Zope because it's rumoured to be unpythonic or force you to do things certain ways, take a long hard look at Zope 3. You'll be happy you did.

"I'm looking for solutions that depend only on the Python standard library, and use accepted Python idioms and patterns. Any takers?"

Yes, Zope 3. :-)

Is internet applications a bad idea?

On newsforge Robin Miller claims that internet applications is a bad idea. He lists three reasons. 1: loss of connection, 2: connection outage and 3, connection failure. OK, so he really mentions only one reason, but he mentions it three times in different variations. Yes, internet connections between X and Y is not 100% reliable. Not only can your hardware or software fail, your connection to the ISP can fail, the connection between your ISP and the application servers ISP can fail and the application servers ISP can fail and the application server can fail and...

Things fail. Connections too. But that is no different from a hardwaree failure or a software crash from the users point of view. So, the more important an application is, the more you need to make sure access to it will work all the time. A rather self-evident point, which Robin Miller fails to reach.

Getting a stable access of course includes minimizing failure points. For internet based applications, which in todays world includes almost anything, since the internet reaches all the way to that 10base-T plug that sticks out the back of your computer, that means you shouldn't have fifteen ISPs between you and the server. Which also seems rather self evident.

The internet today is everywhere, because it facilitates communication in a way we have never seen before. Anything you do via the internet is an internet based application, really. Can you honestly say it is a bad idea?

1.3 of the Calendar released!

If you wonder why it has been so long between 1.0 and 1.3 of the CalCore/CalZope/CPSSharedCalendar trio, then the answer is not only vacation, and a whole bunch of big new features, but also that every time I have been close to releasing a new version, I have first released it to some of our trusty customers for testing. And they have promptly found bugs, which I then fixed, and made a new release, and so on.

After the release of 1.0, several additional customers have installed the new calendar, and we also have people using it outside of CPS altogether. At this point I dare to say that the new calendar is both prettier, faster and more stable than the old CPSCalendar. The type of bugs that has been popping up lately are mostly translation and usability problems, where the calendar doesn't behave exactly as you would expect it to. An upgrade is warmly recommended.

The major new features in 1.3 are:

Major speed increases for big installations.
Event coloring according to type.
Calendar "stacking": You can create calendars which display the events of several calendars as if they were one.
Migration from the older CPSCalendar.
Support for Zope 2.7 and 2.8, CMF 1.4 and 1.5, CPS 3.2 and 3.3.
Localization of date format.

Of course, there are also many bugfixes, most having to do with translations, something that always takes a long time.

Separate releases per target

There are now several separate releases. This is mostly because the install instructions just got way to complicated, and it's also an effect of the desire to release a non-zope release of the CalCore library that does all the heavy calendar lifting, so the rest of teh Python world can use it. CalCore requires some libraries from Zope 3, so I made a CalCore-bundle release that includes these packages, and which is easily installable into any non-zope python installation with the standard distutils.

There is also a CalZope bundle that includes all you need to install it under Zope or CMF (except for Five and Zope 3), which you need if you run Zope 2.7.

And of course, we also have the CPSSharedCalendar bundle, which includes all the products you need to install the calendar on CPS (again not including Five and Zope 3).

You can find the latest version of these bundles on cps-project.com:
http://www.cps-project.org/static/misc/

zpkg tool: A quick intro

You are in a maze of twisty little config files, all alike

I'm currently preparing to create a non-Zope bundle of CalCore and all it's dependencies. Since these dependencies are mostly Zope 3 packages (interface and schema) it seems natural to use Zope Corps tool for this: zpkg.

Unfortunately zpkg is a complex tool with cryptic documentation. I have made packages with zpkg before, and it was hard. This time around, it was equally hard, because I had forgotten everything. I'm sorry to say this, but when it's hard to do something, and hard to remember how you did it, this is a sign of a bad design. Configuring a product to use zpkg is far from easy or logical, and there are many confusingly similar files, all named SOMETHING.cfg.

Getting started

First you need to check it out:
svn co svn://svn.zope.org/repos/main/zpkgtools/tags/zpkg-1.0.0 zpkgtools
or, for the more adventurous:
svn co svn://svn.zope.org/repos/main/zpkgtools/trunk zpkgtools

The main application is in bin/zpkg. It takes many parameters, and after a while I found that the best thing to do is to create a configuration file, and call that with -C. I called mine BUILD.cfg. Zope corp likes to call them the name of the package, such as Zope.cfg, but for me that is confusing, it sounds way to similar to the zope.conf file.

The format of the configuration files the same format as the zope.conf file, that is you use # for comments, and write keyword value pairs, and create groups with <tag> </tag>.

The first keyword I needed was:
collect-dependencies yes
This tells zpkg to include the dependencies of the product that I'm packaging and include them in the package. That's what the whole thing is about, if you don't need that you don't need zpkg at all.

Also, I'd like to not have to write the name of the resource I'm packaging every time I'm packaging, so I include the line
default-collection calcore

However, this will create a package named calcore-x.x.x.tgz. This is confusingly similar to the CalCore-x.x.x.tgz packages created by Nuxeos packaging tool, which includes CalCore only and not dependencies, so I also add
release-name CalCore-bundle
to make the file be called CalCore-bundle-x.x.x.tgz instead.

Now I can run zpkg -C BUILD.cfg, and it will try to build the release. However, it tries to package the calcore resource by default, and has absolutely no idea how to do that, so you get an error.

Defining resources

Resources are defined in a resource map. This is a list of resource names, and a location where you can find them. You can either put the list of resources into the configuration file directly like this:
<resources>
    calcore src/calcore
</resources>
or you can put it in a separate file, lets call it RESOURCES.cfg, like this:
    calcore src/calcore
and just include it in the main file with
    resource-map RESOURCES.cfg

The resource definition for calcore points to a subdirectory in the current hierarchy. In most cases you have the product directly in the same directory as the configuration file you are now creating, and then you would define it up with a dot:
    calcore .

Including the dependencies

At this point, you can now actually make a package. It will however not include the dependencies, which of course is the whole point of this excercise. For this, we need to tell zpkg which dependencies the resource has. This is done with a DEPENDENCIES.cfg file, which simply lists the resources that this resource needs. The calcore resource was defined up as being in src/calcore, so that's where the DEPENDENCIES.cfg needs to be. It looks like this:
zope.interface
zope.schema
zope.i18nmessageid
iCalendar

Of course, zpkg has absolutely no idea where to find these resources, so we have to extend the RESOURCES.cfg file:

zope.interface svn://svn.zope.org/repos/main/Zope3/tags/ZopeX3-3.0.0/src/zope/interface
zope.schema svn://svn.zope.org/repos/main/Zope3/tags/ZopeX3-3.0.0/src/zope/schema
zope.testing svn://svn.zope.org/repos/main/Zope3/tags/ZopeX3-3.0.0/src/zope/testing
zope.i18nmessageid svn://svn.zope.org/repos/main/Zope3/tags/ZopeX3-3.0.0/src/zope/i18nmessageid
iCalendar svn:http://codespeak.net/svn/iCalendar/tag/iCalendar-0.10

Note here that these resources are defined up as svn tags! zpkg will fetch the files directly from the svn of each product. Very practical. You can also have a * instead of the tag-name, in which case it will use some kind of logic to figure out which tag to use (what logic I don't know). Also note that I included zope.testing above. That's because several of the resources above define up their own DEPENDENCIES.cfg, and of course, those dependencies must be defined as well, if they are to be included. zope.testing is really not needed unless you want to run the unittests, but I include it anyway for good measure.

For some reason zpkg will complain that the resource "zope" is not defined and skip it. That's a good thing, because it's would mean that all of Zope3 was included, which is quite silly. Why it wants to include it is beyond me, none of the resources have it in their DEPENDENCIES.cfg file.

Adding files to the distribution root

I want to have some files in the distribution root. A README.txt that tells you what the package is and how to install it should be in every package. I also want the HISTORY file to be included (but named HISTORY.txt), the GPL license, with the name COPYING.txt and the whole doc directory. To add them to the root I need to create a PACKAGE.cfg in the resource directory, that includes the follwing lines:
<load>
HISTORY.txt svn:http://svn.nuxeo.org/pub/CalCore/tags/*/HISTORY
COPYING.txt svn:http://svn.nuxeo.org/pub/CalCore/tags/*/COPYING.txt
README.txt svn:http://svn.nuxeo.org/pub/CalCore/tags/*/README.txt
doc svn:http://svn.nuxeo.org/pub/CalCore/tags/*/doc
</load>

<distribution>
HISTORY.txt
COPYING.txt
README.txt
doc
</distribution>

The first part adds the files into the resource directory. It isn't possible in zpkg to include files that are above the resource root, so I instead load them from svn. This can also be used to have the latest version of the licensing in a separate package and thereby automatically include the correct version every time you package, and other cool things.

The distribution tag also tells zpkg to include these files in the distribution root. (They will end up both there and in the resource directory).

Nice extras

The PUBLICATION.cfg is not needed, but it's a place to put in meta data about the product, which is a good thing:
Name: calcore
Summary: Python calendaring
Home-page: http://www.nuxeo.com/
Author: Martijn Faassen, Lennart Regebro, Nuxeo SARL.
Author-email: lregebro@nuxeo.com
Licence: GPL 2
Description: A python package for making personal and group calendars.

Note that PUBLICATION.cfg uses an RFC-822 type format, that is, it uses keyword-value pairs with a colon after the keyword, while all the other config files does not have a colon.

Thats it!

There is a lot of cfg files involved and it can be very confusing, and I'm in no way near understanding all the details. But at least, after some trial and error, I succeeded in making a tgz that includes what I want.

Zope2 vs Zope3 FAQ

I see many questions on the differences between Zope 2 and Zope 3 on mailing lists. Here is a short attempt to answer some of the most common ones.

Should I use Zope 2 or Zope 3?

Short answer:
If you need a product that runs on Zope 2, like for example CPS, Plone or Silva, you should use Zope 2. Otherwise you should use Zope3.

Long answer:
Zope2 and Zope 3 has slightly different target audiences. Zope 2 was ment as a webserver, where you could create dynamically generated websites and manage them through the web. However, Zope 2 was so powerful and the combination of true OO database and Python so developer friendly, that many people soon started to use it as a web application development platform, creating web applications that had nothing to do with the orginal concept of creating websites.

Zope 3 is directly and originally aimed at that type of application development. That means that is you are looking for a web development platform, you should use Zope 3. If your aim it to make a website, Zope 3 is currently lacking in that department. Mainly, a complete enterprise CMS is not yet available for Zope 3 (at least not as open source), so there is no substitute for CPS or Plone yet.

What about Five?

Five is a technology bridging the gap between Zope 2 and Zope 3. It is included in Zope 2.8, and it enables you to use many of the Zope 3 features in Zope 2. It is therefore mainly of use if you have to use Zope 2 for the above reasons, but you want to start using the nice Zope 3 programming patterns, such as views and adapters.

Currently, products written for Five will NOT run under Zope 3, and products written for Zope 3 will NOT run under Five. However, with every new release of Five and of Zope 2, it is intended to bring Five products and Zope 3 products closer and closer, thereby having a situation where only small modifications, or maybe even no modifications at all, are needed.

Hints for Zope 2 developers

As a Zope 2 developer, it's easy to get into the wrong programming patterns for Zope 3. Many of the techniques used in Zope 2 are not necessary for Zope 3.

Don't use stock objects!

A common pattern in Zope 2 is to monkey-patch the stock objects to provide more functionality. Typical examples of this is that you want to make all folders ordered, so you can change the sorting of the objects, or override all applications manage_afterAdd method to reindex them, and so on.

But Zope 3 is not a way to make websites. It's an application development platform. The application you create are not expected to use any "stock objects". There are not very many of them anyway. You are supposed to create your own objects and use them. This also means you don't have to monkey patch anything. In addition, events and adapters mean that have have better ways of changing and overriding functionality without any monkey patches.

Don't monkey patch!

See above: You shouldn't need to monkey patch in Zope 3. If you do, you are most likely doing something wrong. There is of course the possibility that you need to override something that should be overridable via some sort of configuration, but isn't, in which case this could be seen as a Zope 3 bug. Fixing Zope 3 bugs by monkey patching is OK, provided you also get the bug fix into the next release of Zope 3, of course.

Create your own ZCML statements!

ZCML is the Zope 3 way of configuring. It's only global configuration (configuration done on a site or folder basis needs to be done through the web at the moment) but all that global configuration should go into ZCML. It should absolutely not go into Python code. Why? Because ZCML is overridable. If you have in your product a default ZCML statement to configure your product, the people who use your products do not need to change your code to modify the begaviour of your product. Instead, all they need to do is create a new product with an overrides.zcml, and it will override your statements.

Making ZCML statement may be a bit confusing the first time, but once you got a hang of it, it's quite simple.

Indexing events

I have lately been looking into increasing the performance of CPSSharedCalendar when you have many events in the database (some tens of thousands or more). There are a couple of things that quickly gets tricky when doing this:

No matter what you index on, the result set from each index will be very big.
Indexing recurring events is slightly tricky.

Big result sets

The main thing to filter on for events is obviously start and end time. You create one index for start time and one for end time. When searching you then have one result set with anything that falls before one of these dates and another result set with everything that falls after one of the other dates. And then you need to make a union of these result sets.

Sound simple enough? Well, it is simple, but it is also very slow, because the result sets will be huge, and making a union will take a long time. Creating more indexes does not help, because the only thing you typically search on is the dates and who goes to the meeting, which of course just creates a third huge result set.

The way around this is to only use one index, for the data that is likely to create the smallest result set. This is the index where you compare with start time, as most searches will be on "today" or "this week" and there are typically more events in the past than in the future.

The other data you want to compare with are kept in a sort of "meta data" index, that is, instead of mapping from value to event, you map from event to value. Then you loop through the result set, and fetch the end date value from this "meta data index" and compare with that.

This is faster than having two index result sets and doing a union, because both result sets will be so large that it is faster to get the data event by event, as the amount of comparisons that is needed gets much smaller.

More hints on how to do efficient searching with large results sets are welcome.

Indexing recurring events

The biggest problem of indexing recurring events is that if you set something up to happen every week, forever, you have no end date to index.

I solved this by indexing events with no end date with the value "None" and handling them separately. This causes some extra work. In principle, you need to check every single event that recurs forver in every request. You also need to check every occurence of every recurring event where the time searched falls on the time span of the occurrences.

This can be handled in several ways.

You could index a recurring event with the start date of the first occurrence, and the end date of the last. This is the simplest and slowest.
Or, you could simply index every recurrence separately (except for never ending recurrences, of course).
Or you can mix it. You can index every recurrence for the next year separately, and then index the test as one very long occurrence. You could even index the occurences during the search, using the indexing as a sort of "occurence caching".

But I don't expect recurring events to be that common, so after briefly looking into the faster and more complex ways of recurring this, I went with the simplest and slowest, according to the "You Ain't Gonna Need It" principle. If your calendar becomes slow and this is because you have very many recurring events, then these methods might be of interest.

The results

Well, I'm happy to say that the time to search out a months worth of data amongst a 100.000 events has dropped from 3.5 second to 0.17 seconds thanks to the performance improvements I made last week. I think that should be enough for quite a while.

Extending products Zope3-style.

Today we had a need to extend the CPSSharedCalendar a little bit in a separate project. The functionality needed was not seen as generic enough to go into the CPSSharedCalendar itself. Luckily, since it's written with Five, this can be done easily, and without touching the original code, or monkey patching, and with none of the standard problems of using portal_skins. So I thought it might be interesting to show how this is done, as a way to convert more people to Zope3 and Five.

Basically, the project needed to display a list of events from the users calendar, but only the events from today, and with a custom formatting. Here is the rough steps used, as an example to show to others how to do it:

Create a new product. A folder under Products and an empty __init__.py is all that is needed.
Create a view class that has a method that performs the actions you need. Lets call the module customeventview.py, and the class CustomEventsView.

class CustomEventsView:
    "View for returning a list of all events of today"

    def getEvents(self):
        events = self.context.getTheEventsWhichIwant()
        return events

Create a template that calls the view class, and return the list of events as you want it, lets call it customevents.pt.

<html metal:use-macro="context/@@standard_macros/page">
  <metal:block fill-slot="body">
    <ul tal:define="events view/getEvents">
      <li tal:contents="event/title" />
    </ul>
  </metal:block>
</html>

Create a configure.zcml that connects the template to the view class.

<configure xmlns="http://namespaces.zope.org/zope"
           xmlns:browser="http://namespaces.zope.org/browser">

  <browser:page
    for="calcore.interfaces.ICalendar"
    name="customevents.html"
    template="customevents.pt"
    class=".customeventsview.CustomEventsView"
    permission="calendar.ViewCalendar"
    />

</configure>

This example, as you see, provides no great functionality. Only another, custom view. But the point here is that this customization is completely local, it does not require any modification of the original software, nor does it monkeypatch it. At the same time, should you want to, the customization is easy to move into the original product.

But, you say, this can be done by making a new skin in portal_skins! Sure, but then you run into problems of the restricted code of python scripts. This is all disk based, and has no such problems. Also, normally you would have to set up a custom skin folder for these overrides, and they have to be set up in the Zope site, while this type of Zope3/Five extension only needs you to copy the product into the Products folder and restart Zope. You don't need to run any installation scripts, or add any new skin folder to the list of folders in all the skins.

That's the power of Zope3 and Five.

Advanced security machinery in Zope2

In the new calendar project we have the need for changing the acess to an object dependning on the the attributes of the object. For example, you should not be able to view an event unless one of these conditions are fulfulled:

You are invited to the event
You have the right to view the calendar of somebody who is invited and the event is a public event
You are the manager of somebody invited to the event (even if the event is set as private).

Also, you can only edit the event of you are the event organizer, or the event organizers manager.

There are two basic paths to walk down here. The simplest, and usually the best, is to simply recalcuate the security when you change the event. I chose not to go that path this time, mainly because there are no good hooks to use in pure Zope2. In CPS we have an event system that would work well for this, and the same is true for Zope3, but since the calendar needs to work in pure Zope2 it's not possible. Creating hooks on the events themselves would be possiblem but we would then also need hooks on the calendars to be able to recalculate the security when the security settinsg change on a calendar.

Also, there is a risk that recalculating the security takes a long time of you have many events in the calendar.

So instead, I opted to create the security settings dynamically, when accessed. This requres using a couple of advanced Zope 2 security mechanisms that I think is worth a blog.

Acquisition wrapping

The first "trick" is to make it possible to have security settings on the attendee object. Since the attendee objects are non-persistent objects (they may originate from an external user source, for example) we can't just store the permission settings on the objects as is normal in Zope2. Also, even if we did store the settings in the calendar tool somehow, having a good user interface for this is not trivial.

Therefore, we decided to use the attendees main calendar as a "proxy" for the attendee settings. You give people a role on the calendar, and they will have the same role on the attendee. The result is mainly that you can check access settings directly on the attendee, instead of having to look up the main calendar for the attendee and check there. It greatly simplifies use of the calendar for developers.

The "trick" is simple, we need to make the attendee appear as if it is a subobject of it's own main calendar:

    def getAttendee(self, attendee_id):

        attendee = Attendee(attendee_id, attendee_id, 'INDIVIDUAL')

        homecal = self.getMainCalendarForAttendeeId(attendee_id)

        if homecal is None:

            return attendee.__of__(self)

        # This is to acquire the security settings from the calendar:

        return attendee.__of__(homecal)

It is the __of__ method that wraps the attendee so it looks like a subobject. If the attendee seems to have no homecalendar, it gets the security settings of the calendar tool instead.

We also do the same trickery for events, but for another reason. We in the case of events want to to actually look like, through the user interfaces, as the events are subobject of the calendar. This allows us to have proper crumbtrails and the like. Attendee objects however, have no visible user interface, so for attendeesd this is done only to simplify the security checking on attendees.

An __ac_local_roles__ method

Usually __ac_local_roles__ is a dictionary of user ids and roles. But Zope 2 allows you to make it a method that returns this dictionary instead. So for an event, I can return a dictionary containing all the people and resources attending an event, and giving them the appropriate "EventAttendee" role.

In fact, to return a complete dictionary, I also need to go through all the attendees to see which users have the AttendeeManager role on their calendars, and give them the EventAttendee role to those users as well (see the mentioning above of being somebodys manager). But this can take a very long time if you have many attendees, so I have simplified it. I'll only check the current user, and only the current calendar. That is, even if you have the right to view an event through Martijns calendar, you may not have the right to view the same event through Erics calendar. You also will not be able to check what the permissions another user has. Other users will always look like they have no permissions at all on events. This simplifies the code a lot and speeds up the security handling.

    def __ac_local_roles__(self):

        # Find the current attendee id:

        attendee_src = self.portal_calendar.attendee_source

        current_attendee = attendee_src.getCurrentUserAttendee()

        current_id = current_attendee.getAttendeeId()

        attendees = [current_id]



        # Check if the current user is an attendee manager for this calendar

        current_user = getSecurityManager().getUser()

        calendar = self.getCalendar()

        roles = current_user.getRolesInContext(calendar)

        if 'AttendeeManager' in roles:

            # Current user is attendee manager for this calendars user, so

            # We'll add this calendars user to the list of current attendees

            attendees.append(calendar.getAttendees()[0].getAttendeeId())



        # Check if the current organizer is one of the relevant users, in that

        # case we should get the organizer role as well as local roles:

        if self.getOrganizerId() in attendees:

            roles.append('EventOrganizer')

     

        # Check through the attendees to see if they are participating:

        event_attendees = self.getAttendeeIds()

        for id in attendees:

            if id in event_attendees and 'EventParticipant' not in roles:

                roles.append('EventParticipant')

        return {current_user.getId(): list(roles)}

Dynamic permission proxying

This last technique is probably the least obvious, as it is really a combination of two: Permission proxying and overriding __getattr__.

In Zope2, the roles that have a permission on a particular object is stored in an attribute called "_[name_of_permission]_Permission". For example _View_event_Permission in the case of the permission named "View event". This should be a tuple of roles. In the case of events, we really want the View event to have different sets of roles depending on if the event is public or private. We can make that happen by overriding the events __getattr__-
However returning dictionaries of roles in this case means that the mapping between roles and permissions is fixed, which means that the normal security settings in Zope no longer has any impact on the permission to role mapping for this permission. Breaking standard functionality like that is not a good idea, but luckily there is a method to get around this as well. The permission attribute can be a string instead of a tuple of lists. If it is a string, it is assumed to be the name of another permission, and that permission is used instead. This way we can have three permissions; one for viewing public events, one for viewing private events, and a third "proxy permission" for viewing events that returns the correct permission to use for this event:

    def __getattr__(self, name, default=_marker):

        # View event is a "proxy permission". Nobody actually needs to

        # have that permission, instead, the name of the permission to

        # be used is returned.

        if name == '_View_event_Permission':

            if self.__dict__['access'] == 'PUBLIC':

                return '_View_public_event_Permission'

            else:

                return '_View_private_event_Permission'

        if default is _marker:

            return getattr(self.__dict__, name)

        return getattr(self.__dict__, name, default)

As you see, every time the permission "View event" is accessed, the security machinery is told to look at "View private event" or "View public event", depending on if the event is public or private.

Zope 2.8.0b2 out

I have spent some time this week trying out Zope 2.8 in it's second beta incarnation. There are a lot of basic groundwork changes in Zope 2.8, which is evident from the deluge of deprecation warnings you get when using it with applications like CPS. But other than that, there was only a couple of few things that didn't work straight away, and these have now been fixed.

Zope 2.8 it also includes Five, to get people started on the path to Zope 3 development. This is amazingly cool. The version shipping with Zope 2.8b2 is usable unless you want to internationalize your ZPTs so users can change the interface language. If you want that you need to wait for Five 1.1. Luckily, Zope allows you to easily install a new version of Five in your products directory, and it will use that version instead, so keeping Zope 2.8 up with Five developments will be easy.

All quiet on the Bugtracker front

So far in my life I have used and tried five different bugtracking systems: Bugzilla, Mantis, Zope orgs "Collector", one I wrote myself in Lotus Notes ten years ago, and now lately Trac.

Trac is cool, because it has a Wiki. Other than that, these systems are pretty much all crap, although Trac is quite likely less crap than the others. So this blog started as a complaint over the bad state of bug trackers...

...but, it quickly turned into something less whiney and more constructive; a feature requirement list of bug tracking systems. This are some features I can come up with that I want in a bug tracker. You are welcome to add your own requirements as comments, and if I get a lot of comments, I'll merge this into another blog entry later.

You are also welcome to come with more recommendations of bug trackers. I don't have time to test bug trackers in detail, so if you say bug tracker X has feature Y, I will just believe you. Hype your own product!

1. Workflow

You need to be able to assign tickets to different persons, add comments, reassign and move around, and of course, see what is assigned to you.

2. Flexible schema

Every company/open source community has it's own requirements of what the bug report should include. Lets see some examples of this:

Organizing Projects/Products/Modules/Components.

A company that makes commercial products might want to have products that consist of several component products. For example, Norton SystemWorks include Norton AntiVirus, and Norton DiskDoctor and so on.

Consulting companies need projects that are orthogonal to components.

Therefore there need to be some sort of top-level attribute to each bugreport/ticket (different names, same thing) that selects this product/project, and there must be an attribute to select the component, and these should be orthogonal. A component that is used in several projects/products should not need redefinition in each project/product.

Also, this top level attribute needs to be renamable between project and product (and maybe something else).

Extra candy would be nice an ability to select which modules are used in a project, and a non-orthogonal component collections, or even a component hierarchy. If you have collections of components, these might be called "products", but then again, if your other top hierarchy is "products", they should not. So yet again, the name needs to be configurable. And so should the name "modules" too.

Lets see how the products I tried fulfull this:

Mozilla: Yes! Has a product and a component. These are orthogonal. There is no easy renaming that I know of, but I guess you can change the HTML-forms in worst case. There is no selection of which components are valid for which products, as far as I can see. No non-orthogonal component collections, either, so non of the nice extras are there.

Mantis: No. Has projects and categories but these are not orthogonal. Projects are the all-encompassing major division, and everything else is unique to that project.

Trac: No. Has only components.

Version numbers

Any bug report needs to have a version number field. In a big company, it should be a requiered drop-down box, to make it easier for support to know if this is a known/solved bug that just got reported again. But the numbers in this drop-down box needs to be maintained, and they need to be unique to each product. We all know a small company is not gonna maintain those lists, so it should also be possible to just have a text-field.

Again we see that some configurability here is needed.

Status

What the possible status of a bug report should be causes a lot of discussion when it comes to Zope.orgs collector. This is clearly a matter of different needs. Should the reason for closing be a status or a separate field? depends on what reasons you want to allow, and so on.

I'm sure I could come up with more examples of how the schema needs differs if I want to, but this should be enough.

3. Project management

a. Milestones

Each bug report should be able to be assigned a target milestone and a blocker status (only by the project managers of course). You should be able to get a list of what bugs are blocking, and who the guy is that is supposed to fix it.

Mozilla: Yes!

Mantis: No.

Trac: Yes!

b. XP support

Each bugreport should also be an XP card, and it should be possible to do all the things you do with XP cards, i e set a projected time for how long it probably will take to fix it, as well as set in the actual number of hours it took to fix it when it was fixed. Of course, with the simple addition (whic most bugtrackers already have) to mark something as a feature request, and not a bug, this turns the bugtracker into an XP-project management software.

I know of no software that supports this.

4. Security

The ability to assign different people different security depending on what product/project you are on, and also restricting access to bug reports considering security is a requirement.

5. Good queries with simple access

If you look at one bug ticket, and you want to see the rest for one component, all you should need to do is click the component name. Going to the query page and making a query each time is way to many steps.

I'm so happy I'm using Python

http://www.ozonehouse.com/mark/blog/code/PeriodicTable.html

Python Statement Considered Harmful

Martijn Faassen posted two entries on alternative template languages for Zope 3:

http://faassen.n--tree.net/blog/view/weblog/2005/04/15/0
http://faassen.n--tree.net/blog/view/weblog/2005/04/16/0

The idea behind this is simple: Templates should be data-driven! That is, they should contain no calls, no logic (other than for display) and no other monkey-business at all. It might be easiest to explain why this is a good idea by explaining the problems with NOT doing it.

In the beginning there was DTML

DTML was the first template language for Zope. Although a huge improvement on how PHP and ASP templates work, DTMLs syntax, gets ugly pretty quickly. Also, people tended to write complete aplications in DTML. That's OK; it was designed to do that, Python Scripts didn't exist in the first versions of Zope. DTML was the replacement of PHPand ASP. But this quickly tirned ugly and complicated (as it does in ASP and PHP too). Something better was desired.

Let there be TAL

ZPT was the next step: Better syntax (it's more verbose, but it is consistent, easy and doesn't degrade into uglyness), and less intelligence and magic. Again, a huge step forward. But if you look at the typical CMF template, you'll see masses of logic, usually complex logic that makes the templates hard to understand and error prone.

This happens because templates in Zope2 access objects. They have a context variable where you get the object you are displaying, and you can access the objects attributes and methods. You can also call python expressions. In short, even the new cleaner ZPTs doesn't enforce a strict enough separation between logic and presentation. You can still do too much.

So what if we don't do anything of this? We just have data, and display it? Is that feasible with ZPTs? Yes, absolutely. What you need is to somehow call a method before displaying the template, that fetches all the data and does everything that should be done before displaying.

Keeping your templates clean.

Luckily, we don't need a new templating system to do this, ZPT allows you to do too much, but it doens't force you too. You can do this by convention.

Zope3 has a system of view classes, where the class is supposed to do exactly, this: prepare and react to the data for and from the template. So in Zope 3 you can prepare everything you need, do any actions that should be done and set the data to be displayed as attributes on the view object from the views __init__ method. All in pure, standard python. No longer should you need to call any more methods in the template, and you don't need to do any arithmetic, it's all done beforehand.

In CMF you don't have a view class, but there can call a python script, that returns a dictionary with the data to be displayed, with statements like tal:define="data context/update_script". Then you access it with "data/key" later in the ZPT.

Python statement considered harmful

As a rule of thumb, you should avoid any "python:" statements in ZPT, and you should also avoid using "context" unless it is to use another template via a macro. If you avoid this, you are a long way to keeping your ZPTs clean, and your code understandable.

Workshop: Making music with the internet.

The 4th of May I'll be there talking about making music on the internet; communities, collaboration and distribution, an area where things are moving and changing very rapidly. This talk is a part of a larger event for DJs. The event is held at The Blue Tree in Gaborone, Botswana.

The sickness of Zope

Zope is great, but there are some things that are seriously annoying. Much of it comes from things showing their age. There is lots of code that does things in particular and complicated ways because of the need to behave the way things have always behaved, even if that no longer makes much sense. Also, CMF has a couple of basic design flaws, the biggest of them being portal_skins. So in this way, Zope3 will be very welcome, as it rewrites things from the start, and does thing The Right Way ^(tm).

But there is one thing that is not going to automatically be solved with Zope 3: Peoples tendency in the Zope community to first of all do things their own way, and secondly, not move these "own ways" into the common code base, even if they could be of use to others.

I was hit by one of the more typical examples of this today: I need to have permissions that i check dynamically. Usually in Zope, you protect methods or objects with permissions, using the "security.declareProtected(permission_name, method_name)" call. But sometimes, which permission you want to check is dynamic. This is quite possible to do, but first you then need the permission to exist, so you can set up a role to permission mapping, and then check for the permission based on what roles the user has.

And, it turns out, Zope has no method for creating permissions that are not connected to a method or object or a class. I tried various ways and looked through the code, and found nothing. I turned to Florent, and he pointed out that there is a method to do this. In CMF....

CMF is, just like Zope, created and maintained by Zope Corp. And still, this method, that belongs somewhere in the AccessControl module, is located in CMFCore, where is has no business at all.

The examples of are numerous. The fact that most people use DCWorkflow as the workflow with CMF wasn't reflected until recently, when CMF 1.5 came out, and finally included DCWorkflow. And after much nagging (mostly from us here at Nuxeo) Zope corp decided to include Stefan H Holeks excellent ZopeTestCase product in Zope 2.8, so that is no longer is a pain in the ass to write unit tests for Zope. Another good example are the almost infinite number of user folders that exist. Only last year did Zope corp update come up with a new user folder, instead of the default (and pretty much useless) user folder. The new user folder is PAS, a pluggable, extensible user folder.

And, if you expect it to be included in 2.8....ha! No, it's a separate product. Of course. Meaning that every time somebody needs to use it, they need to send an email to the Zope list and ask which of the different extensible user folders they are supposed to use, since they found at least 4 of the when googling...

This all means that when you use Zope, you have to look all over the place for the little thing that does what you want. There is no coherence, just a sprawling spaghetti, and this in turn has the effect that the learning curve for Zope never ends.

I don't know how to make sure this doesn't happen for Zope3, I guess we all have to just make an extra effort to identify the things that are useful to all, and make sure they end up in a place where everybody can find and use them. codespeak.net has a z3-base which is one effort to have a place for the things that are useful for many, but shouldn't be in Zope3 itself. Having a place like that from the start will at least help a bit. But the rest is up to us.

Moving from Five to Zope3: It gets easier every day.

Well, every week at least. The work on CMF integration was recently released as Five 0.3, and last week the announcement that Zope2.8 will integrate Five and ship with Zope3s component architecture came. So, during the weekend I decided to put the theory to the test: How hard is it to take a product made for Five and moving it to Zope 3? Well, there are both easy bits, and difficult bits. The easy bit are handled by search and replace. They are:

Replacing some Zope2 and Five imports with Zope3 imports.

The difficult bit here is to know what to import. The Zope 2 and Five code isn't organized the same as the Zope 3 code, so "from Products.Five.a import b" is not "from zope.a import b" but rather "from zope.app.foo.browser.a import b". "OFS.Folder" is "zope.app.folder.folder" and so on. So, it's a matter of trying to load Zope3, getting an import error, figuring out what it should be, and doing a search and replace on all python files. Figuring that out is not always easy. For example, Zope2s standard SimpleItem is usually replaces with the two classes Persistent and Contained. That's obvious if you know both Zope 2 and Zope 3 well, but not obvious at all otherwise. Some replacements are almost surprisingly direct, as the folder example above.

Removing all the five-statements from the zcml

Very quick and easy, of course. Few of them have any Zope3 equivalent, so it's typically just a matter of deleting them.

Changing portal_tools into local utilities (CMF only)

This is something that requires more of a rewrite. Not only do you need to rewrite the tools themselves, but all calls to getToolByName needs to be replaced by getUtility to find the tools.

Rewriting any magic

The less magic you use, the less you will have of this. For example, if you do any custom traversal, you have to rewrite that. This will be the trickiest part of any migration, I suspect.

Replacing the CMF macros with zope3 standard macros.

And lastly, this brings us to this weeks improvement. I had to make a search and replace to replace the CMF-type macro inclusion, "here/main_template/macros/master", that I used on all pages, to a Zope3-type inclusion, "context/@@standard_macros/view". But thanks to Sidnei da Silva, who kicked me in the right direction, Five will soon include standard support for the Zope3 style inclusion, and with CMFonFive the Zope3 inclusion will end up inserting the CMF-macro. That way you will probably not have to touch your page templates during the migration. A small step for man, and a rather tiny step for Five, but a step forward nonetheless.

Google does it again.

Do you want your application to do Googe searches? No problem, if you use Python. Just use PyGoogle, dead simple to use Python wrappers for Googles SOAP API. I just downloaded it, got my license key and tested it in just a couple of minutes.

It works, it's full featured, and it's ridicously simple to use. Only problem so far is that you are limited to a 1000 searches a day, at least for the time being, since it's "Beta".

And now for something completely different!

As most readers of Nuxeos blogs will know, Nuxeo organised a sprint here in Paris last week. what started out in December as a sprint to make Five more complete and usable, especially with CMF, had by the time the sprint started turned into an Zope 3 ECM sprint. Mainly thanks to that the necessary Five enhancements had already happened. But after the first day of manu discussions on what we needed, we realized that what we were trying to do was to build more frameworks, but what was needed was more people!

The group of people using Zope 3 is currently small, close-knit and extremely clever. The result of this has been an excellent framework for building web applications. But as we saw with Zope 2, frameworks are not enough. Building something with a framework means you have to learn that framework. What is needed to increase the usage of Zope 3 are finished applications, such as CPS and Plone. But we are not gonna get a finished ECMS application, unless we finish an ECMS framwork. And that will take a long time, if the same small group of Zope 3 developers are to work on it. We need to get more developers working on Zope 3. But for that to happen we need more users of Zope 3. Yes, it's a catch 22.

Luckily there is a way out of that circle of non-usage. We need to make it possible for Zope 2 users to use Zope 3 technologies, in otherwords, we need to make it easy for people to use Five. The decision was therefore to include Five and the relevant parts of Zope3 into the next release of Zope 2. This then became the largest job of the sprint, in one way quite a different direction than was expected, and in another way, just the next step from what we originally intended. Zope 2.8 will have closer Five integration than ever, Five will be included with Zope. This means you as a Zope 2 developer, can start using Zope 3 technologies in your Zope 2 applications, without any extra effort or installation woes.

The aim of being able to make applications that run unmodified both on Zope2 and Zope3 is still quite a bit away. That will be the goal of Zope 2.9.

Five 0.3 released!

Zope3 is very cool, but Zope2 has more products. So, what to do? Well, use both of course. Zope3 + Zope2 equals Five, a product that makes it possible to use Zope3 technologies in Zope 2.

Nuxeo commited to using Five during the fall of 2004, and are soon ready to release the first of our Five based products. But one of the obstacles to this has been Five itself. It hasn't integrated very well with CMF, and therefore not with CPS. Much effort and many different solutions have been tried to make this work, and the final fruit of this work (mainly by Martijn Fassen and me, but with a lot of help, support and ideas from several others) is Five 0.3, released today.

Although isn't possible to just take a Zope3 product and plonk it down into CPS, Five now enables us to use many of the superiour concepts of Zope3 such as adapters and views, and use these for CMF/CPS/Plone development. Zope has always made "rapid application development" possible, but with Zope3 it will be even more so.

CSS quirks

In my current project I have been trying to position things with CSS. This should in theory be easier than using tables, but I still had nagging doubts about it. I was right, of course. It wasn't very easy.

The main lasting problem is that when you set width and height properties in CSS2, this is the width and height of the content of the box, not the box itself. This is contrary to earlier box models, and to be honest, quite unusable. In theory there are many options to solve this, but of course, in practice none of them works.

In my quest for solutions several people (primarily Florent) to redirected me to http://www.quirksmode.org/, which has an excellent overview over most quirks and weirdnesses and solutions for most. Not for this particular problem, however. Internet Explorer for windows simply does not supportany of the solutions I tried, so, until IE gets proper CSS3 support, my boxes will simply have to be slightly too wide on IE for Windows.

Wikinews: A new attitude to news reporting.

Wikinews is quite interesting, even if it is still in it's infancy and many problems need to be sorted. Todays fun news is that I (almost) got quoted in New York Times article on wikinews.

More on User Interfaces.

In a comment to yesterdays rant on the userunfriendliness of Linux Ruslan was nice enough to link to a page on why the default mode of Nautilus is one that opens one window for each folder, something they call spatial mode. It's named after the concept of spatial navigation, which is a good concept.

But as usual, because you name something after a good conept doesn't make it good. Spatial navigation is the idea that we find things in much after where they are located. I'm one of those people who easily can end up with the whole desk covered in paper. Inches deep. Still, I can usually find the papers I want, because I know where on the desk I saw them last, so I only lift half an inch of paper and there I find it. It's all spatial.

So, the idea of a spatial file manager then, according to the Nautilus guys (who in turn got it from John Siracusa) is that each folder has a separate window, and that this window stays in the same place if you close it and re-open it. But of course, that's all completely backwards. What I need to find is the place to click to open the folder. Sure, it's nice if it opens in the same location, but it's unimportant.Spatial location is used to FIND things, and if the location of the open window is the spatial information then the window have to be open for it to give out that information. Of course, a typical unix installation have thousands of folders. Am I supposed to have thousands of windows open?

However, in the explorer-type of file manager (which Nautilus calls "browser"), I have a tree hierarcy to the left, always displayed. And guess what: The folders and files do not move around. They are always sorted in the same alphabetical fashion. The spatial information is there, everytime I open a browser window. I can have ten browser windows open if I want, and they all contain the same spatial information. It's easy and fast to find the directory I want, because it is in the same place it was yesterday.

The spatial mode of nautilus is only useful if you have a very limited set of folders that you always have open. I don't think may people work like that anymore.

Of course, the whole idea of file-systems is an outdated concept from the 60s anyway. But until somebody writes a modern operating system we are stuck with it. ;-)