[Zope-dev] BerkeleyStorage (was RE: [Zope-dev] ZODB and # of objects)

[Phillip J. Eby]

At 10:04 AM 10/22/99 -0400, Michel Pelletier wrote:
>> -----Original Message-----
>> From: Andrew M. Kuchling [mailto:akuchlin@mems-exchange.org]
>> Sent: Thursday, October 21, 1999 9:14 PM
>> To: zope-dev@zope.org
>> Subject: RE: [Zope-dev] ZODB and # of objects
>> 
>> 
>> Michel Pelletier writes:
>> >Are you using substransactions?  If you are using 2.0.x, 
>> then you might
>> >not notice any improvement with sub-transactions because of a
>> >deactivation bug that was fixed just after the release 
>> (sigh).  If you
>> 
>> Trying the CVS version doesn't seem to help; I suspect that the
>> problem is the growing in-memory index of FileStorage objects, and the
>> fix is to use an alternate Storage class that scales better.
>
>Ty has the Berkely Storage, that might work quite well.
>

Keep in mind that this storage is still highly experimental, and also that
it does not currently support Undo or Versions.  (Although Ty might have
Versions done in time for next week's Zope News...  ;) )  But it does offer
superior startup time to FileStorage for large databases and doesn't have
the memory growth overhead that FileStorage does.

On the other hand, BerkeleyStorage does *not* use temporary files for
transaction logging, but keeps everything in memory until the transaction
is committed, so if you have transactions involving large numbers of
objects, Berkeley may not perform as well, memorywise.

The long-term (i.e., a few weeks) plan for BerkeleyStorage is to support
Undo, but only inside of Versions.  We believe this will produce an ideal
trade-off for applications where "run-time" activities would cause undue
file-size growth under FileStorage, but where site
maintainers/admins/content managers still need to have some undo capability.

The other thing we'd like to do is figure out a way to have Versions manage
conflicts optimistically, allowing non-versioned objects to be changed
while a version is being worked on.  This would allow tests of cataloging
things, etc. without locking the catalog outside the version.
ConflictError could then be raised on version commit if the appropriate
transactions aren't undone.  The undo data provided by the version could
include indicators of which transactions involve objects which have been
modified on the 'main line' and need to be undone for commit to occur.

This is a bit more ambitious a project, as it would probably also require
the ability to "revert" individual DB objects to their un-versioned state,
independently of transactional undo, and/or to request a "hard" lock on an
object.