[Zope-DB] BerkeleyDB slow

[Matthew T. Kromer]

Hey name-kin,

You actually wanted zodb-dev@zope.org not zope-db@zope.org.

However, it is true that right now Berkeley DB storages are running 
slower than Data.fs.  There are a couple of factors involved;

1)  Datafs (FileStorage) is a log oriented file -- ie appends just 
happen on the end of the file.  You dont get much faster than just 
appending data.  By the same token, that's why you have to pack the 
database periodically.

2) Berkleley storage is still evolving; in the past few months I've 
heard of several performance tunes, such as increasing the number of 
locks in the storage, etc.

In theory, the Berkeley storage shouldnt be that much slower; most 
updates should happen to shared memory areas and be destaged 
asynchronously.  However, I am not a new Berkeley DB whiz (I used 1.85 
extensively, but thats way old now).  

If I recall correctly (and I may be mistaken) ZEO serially commits 
objects to the Storage.  So, latencies in your writes will tend to 
affect your queue times significantly.   A good question to ask the 
zodb-dev list would be "can ZEO be modified to do concurrent wites for 
some storages?"  I may be barking up the wrong tree, but I think some 
storage methods should be able to support concurrent writes to the 
storage whereas FileStorage can't (since you can't simultaneously append 
without getting bad results).


matt wrote:

> Chris Withers pointed me here to find an answer.  The following is a 
> copy of the emails from the main zope mailing list.
>
>  
>
> Hi,
>
>  
>
> I am finding that writes to the ZODB are far slower when the storage 
> is BerkeleyDB instead of Data.fs.  I should also mention that I run 
> this through ZEO, if that has anything to do with it.  I now have 
> quite a few folders, some with up to 1000 objects in them.  I know ZEO 
> is not very write friendly, but if I switch the Storage back to main 
> .. i.e. Data.fs(Still as a zeo client), and import data into it, then 
> I find it is 5-10 times faster when doing something as simple as 
> editing small dtml method.
>
>  
>
> My current system is the following :
>
>  
>
> Zope Version
>
> (Zope 2.4.0 (source release, python 2.1, linux2), python 2.1.0, freebsd4)
>
> Python Version
>
> 2.1 (#1, Jul 18 2001, 17:06:28) [GCC 2.95.3 [FreeBSD] 20010315 (release)]
>
> System Platform
>
> freebsd4
>
>  
>
> I was wondering if this behaviour was common.
>
>  
>
> regards
>
> Matt
>
>  
>
> ________
>
>  
>
> Reply from Chris Withers
>
>  
>
>
> Indeed. It's two to five times slower.
>
> Ask Barry why (->zodb-dev@zope.org)
>
> cheers
>
> Chris
>
> __________________________
>
>  
>