zpkg tool: A quick intro

You are in a maze of twisty little config files, all alike

I'm currently preparing to create a non-Zope bundle of CalCore and all it's dependencies. Since these dependencies are mostly Zope 3 packages (interface and schema) it seems natural to use Zope Corps tool for this: zpkg.

Unfortunately zpkg is a complex tool with cryptic documentation. I have made packages with zpkg before, and it was hard. This time around, it was equally hard, because I had forgotten everything. I'm sorry to say this, but when it's hard to do something, and hard to remember how you did it, this is a sign of a bad design. Configuring a product to use zpkg is far from easy or logical, and there are many confusingly similar files, all named SOMETHING.cfg.

Getting started

First you need to check it out:
  svn co svn://svn.zope.org/repos/main/zpkgtools/tags/zpkg-1.0.0 zpkgtools
or, for the more adventurous:
  svn co svn://svn.zope.org/repos/main/zpkgtools/trunk zpkgtools

The main application is in bin/zpkg. It takes many parameters, and after a while I found that the best thing to do is to create a configuration file, and call that with -C. I called mine BUILD.cfg. Zope corp likes to call them the name of the package, such as Zope.cfg, but for me that is confusing, it sounds way to similar to the zope.conf file.

The format of the configuration files the same format as the zope.conf file, that is you use # for comments, and write keyword value pairs, and create groups with <tag> </tag>.

The first keyword I needed was:
    collect-dependencies  yes
This tells zpkg to include the dependencies of the product that I'm packaging and include them in the package. That's what the whole thing is about, if you don't need that you don't need zpkg at all.

Also, I'd like to not have to write the name of the resource I'm packaging every time I'm packaging, so I include the line
    default-collection calcore

However, this will create a package named calcore-x.x.x.tgz. This is confusingly similar to the CalCore-x.x.x.tgz packages created by Nuxeos packaging tool, which includes CalCore only and not dependencies, so I also add
    release-name CalCore-bundle
to make the file be called CalCore-bundle-x.x.x.tgz instead.

Now I can run zpkg -C BUILD.cfg, and it will try to build the release. However, it tries to package the calcore resource by default, and has absolutely no idea how to do that, so you get an error.

Defining resources

Resources are defined in a resource map. This is a list of resource names, and a location where you can find them. You can either put the list of resources into the configuration file directly like this:
  <resources>
    calcore src/calcore
  </resources>
or you can put it in a separate file, lets call it RESOURCES.cfg, like this:
    calcore src/calcore
and just include it in the main file with
    resource-map RESOURCES.cfg

The resource definition for calcore points to a subdirectory in the current hierarchy. In most cases you have the product directly in the same directory as the configuration file you are now creating, and then you would define it up with a dot:
    calcore .

Including the dependencies

At this point, you can now actually make a package. It will however not include the dependencies, which of course is the whole point of this excercise. For this, we need to tell zpkg which dependencies the resource has. This is done with a DEPENDENCIES.cfg file, which simply lists the resources that this resource needs. The calcore resource was defined up as being in src/calcore, so that's where the DEPENDENCIES.cfg needs to be. It looks like this:
  zope.interface
  zope.schema
  zope.i18nmessageid
  iCalendar

Of course, zpkg has absolutely no idea where to find these resources, so we have to extend the RESOURCES.cfg file:

  zope.interface svn://svn.zope.org/repos/main/Zope3/tags/ZopeX3-3.0.0/src/zope/interface
  zope.schema svn://svn.zope.org/repos/main/Zope3/tags/ZopeX3-3.0.0/src/zope/schema
  zope.testing svn://svn.zope.org/repos/main/Zope3/tags/ZopeX3-3.0.0/src/zope/testing
  zope.i18nmessageid svn://svn.zope.org/repos/main/Zope3/tags/ZopeX3-3.0.0/src/zope/i18nmessageid
  iCalendar svn:http://codespeak.net/svn/iCalendar/tag/iCalendar-0.10

Note here that these resources are defined up as svn tags! zpkg will fetch the files directly from the svn of each product. Very practical. You can also have a * instead of the tag-name, in which case it will use some kind of logic to figure out which tag to use (what logic I don't know). Also note that I included zope.testing above. That's because several of the resources above define up their own DEPENDENCIES.cfg, and of course, those dependencies must be defined as well, if they are to be included. zope.testing is really not needed unless you want to run the unittests, but I include it anyway for good measure.

For some reason zpkg will complain that the resource "zope" is not defined and skip it. That's a good thing, because it's would mean that all of Zope3 was included, which is quite silly. Why it wants to include it is beyond me, none of the resources have it in their DEPENDENCIES.cfg file.

Adding files to the distribution root

I want to have some files in the distribution root. A README.txt that tells you what the package is and how to install it should be in every package. I also want the HISTORY file to be included (but named HISTORY.txt), the GPL license, with the name COPYING.txt and the whole doc directory. To add them to the root I need to create a PACKAGE.cfg in the resource directory, that includes the follwing lines:
<load>
  HISTORY.txt svn:http://svn.nuxeo.org/pub/CalCore/tags/*/HISTORY
  COPYING.txt svn:http://svn.nuxeo.org/pub/CalCore/tags/*/COPYING.txt
  README.txt  svn:http://svn.nuxeo.org/pub/CalCore/tags/*/README.txt
  doc         svn:http://svn.nuxeo.org/pub/CalCore/tags/*/doc
</load>

<distribution>
  HISTORY.txt
  COPYING.txt
  README.txt
  doc
</distribution>

The first part adds the files into the resource directory. It isn't possible in zpkg to include files that are above the resource root, so I instead load them from svn. This can also be used to have the latest version of the licensing in a separate package and thereby automatically include the correct version every time you package, and other cool things.

The distribution tag also tells zpkg to include these files in the distribution root. (They will end up both there and in the resource directory).

Nice extras

The PUBLICATION.cfg is not needed, but it's a place to put in meta data about the product, which is a good thing:
  Name: calcore
  Summary: Python calendaring
  Home-page: http://www.nuxeo.com/
  Author: Martijn Faassen, Lennart Regebro, Nuxeo SARL.
  Author-email: lregebro@nuxeo.com
  Licence: GPL 2
  Description: A python package for making personal and group calendars.

Note that PUBLICATION.cfg uses an RFC-822 type format, that is, it uses keyword-value pairs with a colon after the keyword, while all the other config files does not have a colon.

Thats it!

There is a lot of cfg files involved and it can be very confusing, and I'm in no way near understanding all the details. But at least, after some trial and error, I succeeded in making a tgz that includes what I want.