Julien Anguenot

embeddable document management core

repoA /

          P /

             trunk /

             branches /

             tags /

$ svnadmin dump --quiet --revision 28000:38000 /home/svn/repoA > repoA.dump

$ svnadmin dump --help

dump: usage: svnadmin dump REPOS_PATH [-r LOWER[:UPPER] ] [--incremental]

Dump the contents of filesystem to stdout in a 'dumpfile'

portable format, sending feedback to stderr.  Dump revisions

LOWER rev through UPPER rev.  If no revisions are given, dump all

revision trees.  If only LOWER is given, dump that one revision tree.

If --incremental is passed, then the first revision dumped will be

a diff against the previous revision, instead of the usual fulltext.

Valid options:

  -r [--revision] arg      : specify revision number ARG (or X:Y range)

  --incremental            : dump incrementally

  --deltas                 : use deltas in dump output

  -q [--quiet]             : no progress (only errors) to stderr

$ cat repoA.dump | svndumpfilter include P > P.dump

$ svndumpfilter --help

general usage: svndumpfilter SUBCOMMAND [ARGS & OPTIONS ...]

Type 'svndumpfilter help <subcommand>' for help on a specific subcommand.

Available subcommands:

   exclude

   include

   help (?, h)

$ svnadmin load --quiet /home/svn/repoB < P.dump

$ svnadmin load --help

load: usage: svnadmin load REPOS_PATH

Read a 'dumpfile'-formatted stream from stdin, committing

new revisions into the repository's filesystem.  If the repository

was previously empty, its UUID will, by default, be changed to the

one specified in the stream.  Progress feedback is sent to stdout.

Valid options:

  -q [--quiet]             : no progress (only errors) to stderr

  --ignore-uuid            : ignore any repos UUID found in the stream

  --force-uuid             : set repos UUID to that found in stream, if any

  --use-pre-commit-hook    : call pre-commit hook before committing revisions

  --use-post-commit-hook   : call post-commit hook after committing revisions

  --parent-dir arg         : load at specified directory in repository

$ svn remove http://svn.foo.com/repoA/P

$ svn move https://svn.foo.com/repoA/P https://svn.foo.com/repoB/

  >>> import cElementTree as etree

  >>> stream = """<?xml version="1.0" encoding="UTF-8" ?>

  ... <doc xmlns="http://bar"

  ...      xmlns:foo="http://foo/">

  ...   <foo:sub/>

  ... </doc>""

  >>>

  >>> doc = etree.XML(stream)

  >>> print etree.tostring(doc, encoding="UTF-8")

  <?xml version="1.0" encoding="UTF-8" ?>

  <ns0:doc xmlns:ns0="http://bar">

    <ns1:sub xmlns:ns1="http://foo" />

  </ns0:doc>

  >>>

  >>> import cElementTree

  >>> import elementtree.ElementTree

  >>>

  >>> my_namespaces = {'http://foo' : 'foo',

  ...                  'http://bar/' : bar}

  >>> elementtree.ElementTree._namespace_map.update(my_namespaces)

  >>>

  >>> stream = """<?xml version="1.0" encoding="UTF-8" ?>

  ... <doc xmlns="http://bar"

  ...      xmlns:foo="http://foo">

  ... <foo:sub/>

  ... </doc>"""

  >>>

  >>> doc = cElementTree.XML(stream)

  >>> print cElementTree.tostring(doc)

  <bar:doc xmlns="http://bar">

    <foo:sub xmlns:foo="http://foo" />

  </bar:doc>

  _namespace_map = {

      # "well-known" namespace prefixes

      "http://www.w3.org/XML/1998/namespace": "xml",

      "http://www.w3.org/1999/xhtml": "html",

      "http://www.w3.org/1999/02/22-rdf-syntax-ns#": "rdf",

      "http://schemas.xmlsoap.org/wsdl/": "wsdl",

  }

OOo_NS = "http://openoffice.org/2000/"

OFFICE_NS = "%soffice" % OOo_NS

TABLE_NS  = "%stable" % OOo_NS

STYLE_NS  = "%sstyle" % OOo_NS

TEXT_NS   = "%stext" % OOo_NS

META_NS   = "%smeta" % OOo_NS

SCRIPT_NS = "%sscript" % OOo_NS

DRAWING_NS = "%sdrawing" % OOo_NS

CHART_NS = "%schart" % OOo_NS

NUMBER_NS = "%snumber" % OOo_NS

DATASTYLE_NS = "%sdatastyle" % OOo_NS

DR3D_NS = "%sdr3d" % OOo_NS

FORM_NS = "%sform" % OOo_NS

CONFIG_NS = "%sconfig" % OOo_NS

FO_NS = "http://www.w3.org/1999/XSL/Format"

XLINK_NS = "http://www.w3.org/1999/xlink"

SVG_NS =  "http://www.w3.org/2000/svg"

MATH_NS = "http://www.w3.org/1998/Math/MathML"

# This will be used for the XML serialization and elementtree.

NAMESPACE_MAP = {

    OFFICE_NS    : 'office',

    TABLE_NS     : 'table',

    STYLE_NS     : 'style',

    TEXT_NS      : 'text',

    META_NS      : 'meta',

    SCRIPT_NS    : 'script',

    DRAWING_NS   : 'drawing',

    CHART_NS     : 'chard',

    NUMBER_NS    : 'number',

    DATASTYLE_NS : 'datastyle',

    DR3D_NS      : 'dr3d',

    FORM_NS      : 'form',

    CONFIG_NS    : 'config',

    MATH_NS      : 'math',

    SVG_NS       : 'svg',

    XLINK_NS     : 'xlink',

    FO_NS        : 'fo',

   }

import elementtree.ElementTree as etree

etree._namespace_map.update(NAMESPACE_MAP)

      >>> def addAfterCommitHook(hook, args=(), kws=None):

      ...    """Register a hook to call after a transaction commit attempt.

      ...

      ...    he specified hook function will be called after the transaction

      ...    commit succeeds or aborts. The first argument passed to the hook

      ...    is a Boolean value, true if the commit succeeded, or false if the

      ...    commit aborted.  `args` specifies additional positional, and `kws`

      ...    keyword, arguments to pass to the hook.  `args` is a sequence of

      ...    positional arguments to be passed, defaulting to an empty tuple

      ...    (only the true/false success argument is passed).  `kws` is a

      ...    dictionary of keyword argumet names and values to be passed, or

      ...    the default None (no keyword arguments are passed).

      ...

      ...    Multiple hooks can be registered and will be called in the order they

      ...    were registered (first registered,first called).  This method can

      ...    also be called from a hook an executing hook can register more

      ...    hooks.  Applications should take care to avoid creating infinite loops

      ...    by recursively registering hooks.

      ...

      ...    Hooks are called only for a top-level commit.  A subtransaction

      ...    commit or savepoint creation does not call any hooks.  Calling a

      ...    hook "consumes" its registration:  hook registrations do not

      ...    persist across transactions.  If it's desired to call the same

      ...    hook on every transaction commit, then addAfterCommitHook() must be

      ...    called with that hook during every transaction; in such a case

      ...    consider registering a synchronizer object via a TransactionManager's

      ...    registerSynch() method instead.

      ...    """

      >>>

  Let's define a hook to call, and a way to see that it was called.

    >>> log = []

    >>> def reset_log():

    ...     del log[:]

    >>> def hook(status, arg='no_arg', kw1='no_kw1', kw2='no_kw2'):

    ...     log.append("%r arg %r kw1 %r kw2 %r" (status, arg, kw1, kw2))

  Now register the hook with a transaction.

    >>> import transaction

    >>> t = transaction.begin()

    >>> t.addAfterCommitHook(hook, '1')

  We can see that the hook is indeed registered.

    >>> [(hook.func_name, args, kws)

    ...  for hook, args, kws in t.getAfterCommitHooks()]

    [('hook', ('1',), {})]

  When transaction commit is done, the hook is called, with its arguments.

    >>> log

    []

    >>> t.commit()

    >>> log

    ["True arg '1' kw1 'no_kw1' kw2 'no_kw2'"]

    >>> reset_log()

  A hook's registration is consumed whenever the hook is called. 
  Since the hook above was called, it's no longer registered:

    >>> len(list(t.getAfterCommitHooks()))

    0

    >>> transaction.commit()

    >>> log

    []

  The hook is only called after a full commit, not for a savepoint or subtransaction.

    >>> t = transaction.begin()

    >>> t.addAfterCommitHook(hook, 'A', dict(kw1='B'))

    >>> dummy = t.savepoint()

    >>> log

    []

    >>> t.commit(subtransaction=True)

    >>> log

    []

    >>> t.commit()

    >>> log

    ["True arg 'A' kw1 'B' kw2 'no_kw2'"]

    >>> reset_log()

  If a transaction is aborted, no hook is called.

    >>> t = transaction.begin()

    >>> t.addAfterCommitHook(hook, ["OOPS!"])

    >>> transaction.abort()

    >>> log

    []

    >>> transaction.commit()

    >>> log

    []

  The hook is called after the commit is done, so even if the

  commit fails the hook will have been called.  To provoke failures in

  commit, we'll add failing resource manager to the transaction.

    >>> class CommitFailure(Exception):

    ...     pass

    >>> class FailingDataManager:

    ...     def tpc_begin(self, txn, sub=False):

    ...         raise

    ...     def abort(self, txn):

    ...         pass

    >>> t = transaction.begin()

    >>> t.join(FailingDataManager())

    >>> t.addAfterCommitHook(hook, '2')

    >>> t.commit()

    Traceback (most recent call last):

    ...

    CommitFailure

    >>> log

    ["False arg '2' kw1 'no_kw1' kw2 'no_kw2'"]

    >>> reset_log()

  Let's register several hooks.

    >>> t = transaction.begin()

    >>> t.addAfterCommitHook(hook, '4', dict(kw1='4.1'))

    >>> t.addAfterCommitHook(hook, '5', dict(kw2='5.2'))

  They are returned in the same order by getAfterCommitHooks.

    >>> [(hook.func_name, args, kws)  #doctest: +NORMALIZE_WHITESPACE

    ...  for hook, args, kws in t.getAfterCommitHooks()]

    [('hook', ('4',), {'kw1': '4.1'}),

     ('hook', ('5',), {'kw2': '5.2'})]

  And commit also calls them in this order.

    >>> t.commit()

    >>> len(log)

    2

    >>> log  #doctest: +NORMALIZE_WHITESPACE

    ["True arg '4' kw1 '4.1' kw2 'no_kw2'",

     "True arg '5' kw1 'no_kw1' kw2 '5.2'"]

    >>> reset_log()

  While executing, a hook can itself add more hooks, and they will all

  be called before the real commit starts.

    >>> def recurse(status, txn, arg):

    ...     log.append('rec' + str(arg))

    ...     if arg:

    ...          txn.addAfterCommitHook(hook, '-')

    ...          txn.addAfterCommitHook(recurse, (txn, arg-1))

    >>> t = transaction.begin()

    >>> t.addAfterCommitHook(recurse, (t, 3))

    >>> transaction.commit()

    >>> log  #doctest: +NORMALIZE_WHITESPACE

    ['rec3',

             "True arg '-' kw1 'no_kw1' kw2 'no_kw2'",

     'rec2',

             "True arg '-' kw1 'no_kw1' kw2 'no_kw2'",

     'rec1',

             "True arg '-' kw1 'no_kw1' kw2 'no_kw2'",   

     'rec0']

    >>> reset_log()

  If an after commit hook is raising an exception then it will log a

  message at error level so that if other hooks are registered they

  can be executed. We don't support execution dependencies at this
  level.

    >>> mgr = transaction.TransactionManager()

    >>> do = DataObject(mgr)

    >>> def hookRaise(status, arg='no_arg', kw1='no_kw1',  kw2='no_kw2'):

    ...     raise TypeError("Fake raise")

    >>> t = transaction.begin()

    >>> t.addAfterCommitHook(hook, ('-', 1))

    >>> t.addAfterCommitHook(hookRaise, ('-', 2))

    >>> t.addAfterCommitHook(hook, ('-', 3))

    >>> transaction.commit()

    >>> log

    ["True arg '-' kw1 1 kw2 'no_kw2'", "True arg '-' kw1 3 kw2 'no_kw2'"]

    >>> reset_log()

  Test that the associated transaction manager has been cleanup when

  after commit hooks are registered

    >>> mgr = transaction.TransactionManager()

    >>> do = DataObject(mgr)

    >>> t = transaction.begin()

    >>> len(t._manager._txns)

    1

    >>> t.addAfterCommitHook(hook, ('-', 1))

    >>> transaction.commit()

    >>> log

    ["True arg '-' kw1 1 kw2 'no_kw2'"]

    >>> len(t._manager._txns)

    0

    >>> reset_log()

  The transaction is already committed when the after commit hooks

  will be executed. Executing the hooks must not have further

  effects on persistent objects.

  Start a new transaction

    >>> t = transaction.begin()

  Create a DB instance and add a IOBTree within

    >>> from ZODB.tests.util import DB

    >>> from ZODB.tests.util import P

    >>> db = DB()

    >>> con = db.open()

    >>> root = con.root()

    >>> root['p'] = P('julien')

    >>> p = root['p']

    >>> p.name

    'julien'

  This hook will get the object from the `DB` instance and change the flag attribute.

    >>> def badhook(status, arg=None, kw1='no_kw1',  kw2='no_kw2'):

    ...     p.name = 'jul'

  Now register this hook and commit.

    >>> t.addAfterCommitHook(badhook, (p, 1))

    >>> transaction.commit()

  Nothing should have changed since it should have been aborted.

    >>> p.name

    'julien'

    >>> db.close()

<?xml version="1.0" encoding="UTF-8"?>

<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema">

  <xs:attribute name="title" use="required" type="xs:normalizedString" />

  <xs:attribute name="description" use="required" type="xs:string"/>

</xs:schema>

import zope.interface

import zope.xmlschema

class IFoo(zope.interface.Interface):

    zope.xmlschema.set('foo')

import persistent

import zope.interface

from zope.xmlschema.demo.interfaces import IFoo

class Foo(persistent.Persistent):

    zope.interface.implements(IFoo)

    def __init__(self, title='', description=''):

        self.title = title

        self.description = description

<xmlschema

      id = "foo"

      document ="foo.xs"

      >

  </xmlschema>

 <content class=".foo.Foo">

    <implements

  interface="zope.app.annotation.IAttributeAnnotatable" />

    <require permission="zope.View"

            interface=".interfaces.IFoo" />

    <require permission="zope.ManageContent"

       set_schema=".interfaces.IFoo" />

  </content>

<browser:addform

      schema=".interfaces.IFoo"

      label="Add a Foo content object"

      content_factory=".foo.Foo"   

      arguments="title description"

      name="addFoo.html"

      permission="zope.ManageContent"

      />

  <browser:editform

      schema=".interfaces.IFoo"

      label="Edit Foo"

      name="edit.html"

      menu="zmi_views" title="Edit"

      permission="zope.ManageContent"

      />

<browser:addMenuItem

      class=".foo.Foo"

      title="Foo"

      permission="zope.ManageContent"

      view="addFoo.html"

      />

<configure xmlns="http://namespaces.zope.org/zope">

    <XMLSchema

       id="sample"

         document="src/zope/xmlschema/tests/examples/simple.xsd"

      />

  </configure>

  >>> import zope.interface

  >>> import zope.xmlschema

  >>> class IFoo(zope.interface.Interface):

  ...    zope.xmlschema.set('sample')

filepath is a path to an XML Schema definition as defined below :

<?xml version="1.0" encoding="UTF-8"?>

<xsd:schema xmlns:xsd="http://www.w3.org/2001/XMLSchema">

<xsd:element name="title" type="xsd:string"/>

<xsd:element name="description" type="xsd:string"/>

</xsd:schema>

    >>> import zope.interface

    >>> import zope.xmlschema

    >>> class ITest(zope.interface.Interface):

    ...     zope.xmlschema.set(filepath)

Now introspect the fields defined within the xsd. We will see two new

fields : title and description.

    >>> names = list(ITest)

    >>> names.sort()

    >>> names

    ['description', 'title']

Now define a class implementing this interface.

    >>> class Test(object):

    ...     zope.interface.implements(ITest)

    ...     title = ''

    ...     description = ''

    >>>

ITest is implemented by Test

    >>> ITest.implementedBy(Test)

    True

Let's create an instance of Test

    >>> test = Test()

Defines potential values for the two fields

    >>> title = u"Title field"

    >>> description = u"Description field"

Now we get the fields from the interface:

  >>> title_field = ITest.get('title')

  >>> description_field = ITest.get('description')

Next we have to bind these fields to the context, so that instance-specific

information can be used for validation:

    >>> title_bound = title_field.bind(test)

    >>> description_bound = description_field.bind(test)

Now try to validate them :

    >>> title_bound.validate(title)

    >>> description_bound.validate(description)

    http://cvs.nuxeo.org/cgi-bin/viewcvs.cgi/CPS3/CPSCore/interfaces/IZODBTransactionHook.py?rev=1.1&view=auto

Pre-commit hook definition with subscribers :

http://cvs.nuxeo.org/cgi-bin/viewcvs.cgi/CPS3/CPSCore/PatchZODB.py

Subscriber definition :

http://cvs.nuxeo.org/cgi-bin/viewcvs.cgi/CPS3/CPSCore/TransactionCommitSubscribers.py?rev=1.1&view=auto

Subscriber interface :

http://cvs.nuxeo.org/cgi-bin/viewcvs.cgi/CPS3/CPSCore/interfaces/ITransactionCommitSubscriber.py?rev=1.1&view=auto

You may register your own subscribers by implementing the interface

above and register them to the list of pre-commit transaction subscribers.

Julien Anguenot

ICSSEA 2006 : Nuxeo 5 talk this thursday

Java soon to be opened !

Nuxeo Core 1.0 released !

Switching from Emacs to PyDev...

EuroPython 2006 slides : A Lucene-based cataloging solution for Zope-3 and CPS-3.4

EuroPython 2006 : timetable of the conference partially announced

St�fane just switched to Mac !

Lucene-based cataloging solution for Zope-3 and CPS-3.4

Motivations

What is Lucene ?

What is PyLucene ?

NXLucene : standalone Lucene indexation server

nuxeo.lucene : Zope 3 cataloging component

CPSLuceneCatalog : CMF Catalog replacement for CPS-3. 4

Already significant results !

Looking for support ?

Thanks

Slides of my Zope3 talk at the www2006 in Edinburgh

Dump and load foreign subversion repositories using svnadmin and svndumpfilter

What for ?

Dumping the repository

Filtering P component from repoA's dump

Loading back P dump to repoB

Zope3 talk at the 15th World Wide Web international conference

ElementTree, serialization and namespace prefixes

Looking for fast and memory friendly Python XML processing ?

Thank you Fredrik Lundh. You're the man !

What about lxml ?

My slides from Solution Linux 2006 about ECM and Zope3

Zope 3.2 and ZODB 3.6 final releases are out !

ZODB 3.7 : after commit hooks

Motivations

Implementation details

Examples

CPS Platform R&D paper from EuroPython 2005 has been published

blogs @ nuxeo.com updated !

CPSGeo : Simple GIS for CPS

What is it ?

Documentation

Download

Screenshots

Report bugs

Home page

"z3ecm roadmap" slides from the Nuxeo technical kickoff 2005

XML Schema support on Zope3 (Concret example)

XML Schema support on Zope3 (update #1)

XML Schema support on Zope3

Interfaces versus ABC within Python

Zope3.1 -- Z3/ECM status and future

RC1 of Zope 3.1 released !

luma LDAP client : PyQT based UI

Promising Zope-3.2 and Twisted integration

Zope-3.1 bug day on Friday 29th for the first 3.1 RC

RMLL2005 : My slides from the Python track are available

Python doctest mode with emacs

Zope3 / ECM -- EP2005 sprint : ecmworkflow and architecture draft

Zope-3.1 beta1 released !

Zope3 / ECM : Project launched !

That's official ! The new Zope3 / ECM project is launched !

What is Z3Lab ?

ECM Platform

GNUpod : iPod on a Linux box

On the road to CPS-3.4: CPSPortlets / CPSSkins shipped within the standard CPS distribution

CPSCore adds pre-commit transaction hooks support on ZODB 3.2

Activity based workflows vs stateful workfows

Paris sprint day #1

YAGNI