[Zope-Annce] ZODB 3.2 beta 1 released

[Jeremy Hylton]

I'm pleased to announce the release of ZODB 3.2 beta 1.  We have
completed all the new features planned for ZODB 3.2 and have wrung out a
number of bugs.  I expect this release is still a bit rough, so I expect
to do another beta release.

ZODB 3.2 has a number of new features and improvements over ZODB 3.1:
  - improve performance and stability of ZEO
  - new ZEO authentication protocol
  - new configuration language, ZConfig, for databases, storages, 
    and ZEO servers
  - many bug fixes

This release of ZODB has been tested with Python 2.1.3, 2.2.3, and
2.3b1.  It should also be compatible with Zope 2.6; Zope users should be
able to install the new ZODB code on top of an existing Zope 2.6
installation.

You can download the source release and Windows installers for 2.1 and
2.2 from http://www.zope.org/Products/ZODB3.2.

I've included the most recent entries from NEWS.txt below.

Jeremy

What's new in ZODB3 3.2 beta 1
==============================
Release date: 30-May-2003

ZODB
----

Invalidations are now processed atomically.  Each transaction will see
all the changes caused by an earlier transaction or none of them.
Before this patch, it was possible for a transaction to see invalid
data because it saw only a subset of the invalidations.  This is the
most likely cause of reported BTrees corruption, where keys were
stored in the wrong bucket.  When a BTree bucket splits, the bucket
and the bucket's parent are both modified.  If a transaction sees the
invalidation for the bucket but not the parent, the BTree in memory
will be internally inconsistent and keys can be put in the wrong
bucket.  The atomic invalidation fix prevents this problem.

A number of minor reference count fixes in the object cache were
fixed.  That's the cPickleCache.c file.

It was possible for a transaction that failed in tpc_finish() to lose
the traceback that caused the failure.  The transaction code was fixed
to report the original error as well as any errors that occur while
trying to recover from the original error.

The "other" argument to copyTransactionsFrom() only needs to have an
.iterator() method.  For convenience, change FileStorage's and
BDBFullStorage's iterator to have this method, which just returns
self.

Mount points are now visible from mounted objects.

Fixed memory leak involving database connections and caches.  When a
connection or database was closed, the cache and database leaked,
because of a circular reference involving the cache.  Fixed the cache
to explicitly clear out its contents when its connection is closed.

The ZODB cache has fewer methods.  It used to expose methods that
could mutate the dictionary, which allowed users to violate internal
invariants. 

ZConfig
-------

It is now possible to configure ZODB databases and storages and ZEO
servers using ZConfig.

ZEO & zdaemon
-------------

ZEO now supports authenticated client connections.  The default
authentication protocol uses a hash-based challenge-response protocol
to prove identity and establish a session key for message
authentication.  The architecture is pluggable to allow third-parties
to developer better authentication protocols.

There is a new HOWTO for running a ZEO server.  The draft in this
release is incomplete, but provides more guidance than previous
releases.  See the file Doc/ZEO/howto.txt.


The ZEO storage server's transaction timeout feature was refactored
and made slightly more rebust.

A new ZEO utility script, ZEO/mkzeoinst.py, was added.  This creates a
standard directory structure and writes a configuration file with
mostly default values, and a bootstrap script that can be used to
manage and monitor the server using zdctl.py (see below).

Much work was done to improve zdaemon's zdctl.py and zdrun.py scripts.
(In the alpha 1 release, zdrun.py was called zdaemon.py, but
installing it in <prefix>/bin caused much breakage due to the name
conflict with the zdaemon package.)  Together with the new
mkzeoinst.py script, this makes controlling a ZEO server a breeze.

A ZEO client will not read from its cache during cache verification.
This fix was necessary to prevent the client from reading inconsistent
data.

The isReadOnly() method of a ZEO client was fixed to return the false
when the client is connected to a read-only fallback server.

The sync() method of ClientStorage and the pending() method of a zrpc
connection now do both input and output.

The short_repr() function used to generate log messages was fixed so
that it does not blow up creating a repr of very long tuples.

Storages
--------

FileStorage has a new pack() implementation that fixes several
reported problems that could lead to data loss.

Two small bugs were fixed in DemoStorage.  undoLog() did not handle
its arguments correctly and pack() could accidentally delete objects
created in versions.

Fixed trivial bug in fsrecover that prevented it from working at all.

FileStorage will use fsync() on Windows starting with Python 2.2.3.

FileStorage's commit version was fixed.  It used to stop after the
first object, leaving all the other objects in the version.

BTrees
------

Trying to store an object of a non-integer type into an IIBTree
or OIBTree could leave the bucket in a variety of insane states.  For
example, trying

    b[obj] = "I'm a string, not an integer"

where b is an OIBTree.  This manifested as a refcount leak in the test
suite, but could have been much worse (most likely in real life is that
a seemingly arbitrary existing key would "go missing").

When deleting the first child of a BTree node with more than one
child, a reference to the second child leaked.  This could cause
the entire bucket chain to leak (not be collected as garbage
despite not being referenced anymore).

Other minor BTree leak scenarios were also fixed.

Tools
-----

New tool zeoqueue.py for parsing ZEO log files, looking for blocked
transactions.

New tool repozo.py (originally by Anthony Baxter) for performing
incremental backups of Data.fs files.

The fsrecover.py script now does a better job of recovering from
errors the occur in the middle of a transaction record.  Fixed several
bugs that caused partial or total failures in earlier versions.