[Zope-dev] Re: ZODB3 Database question/server tuning

[Michel Pelletier]

> -----Original Message-----
> From: Jason Spisak [mailto:webmaster@mtear.com]
> Sent: Tuesday, July 06, 1999 12:44 PM
> To: zope-dev@zope.org
> Subject: [Zope-dev] Re: ZODB3 Database question/server tuning
> 
> 
> Zopists,
> 
> I'm not sure who to ask this to so I'll throw it out and see 
> if anybody
> knows.  
> 
> As far as disk/server performance and the Data.bbb file, it's 
> not like a
> traditional RDB as far as disk performance and speed goes right? 
> Assuming I'm not connecting to any backend DB's, is it just one large
> file always in the same place?  With the advent of ZODB3 how does Zope
> handle many people asking for different protions of the database? 
> 
> (I don't think my question is clear yet, I'll keep trying:)
> 
> Does Zope store the entire Data.bbb file into memory if it can?

Zope stores recently accessed objects in memory.  It deactivates them
according to how you tune the cache in the Control Panel.

>  In
> which case lots od RAM matters and hard disk speed doesn't accept for
> start-up.  Are there lots of random reads/writes to disk if 
> say 150 are
> all creating records at the same time and searching, etc...?  
> Or are the
> writes to disk sequencial because it's only one file (Data.bbb)?  

All writes to the Data.bbb file are appended onto the end of the file.
Note that even though Zope 'writes' something 'out to file' the OS may
in fact be buffering large amounts of the reads and writes in memory.
Linux, for example, will use as much available RAM as it can for file
buffers and caches.  Other UNIXish OSes probably behave similarly.  The
more RAM you have, the less actual disk reading and writing goes on.

Finding and searching a catalog are two different access paterns.
Finding walks down the Zope tree examining objects for certain
attributes.  This is similar to the UNIX 'find' utility.  It is slow and
rather expensive.

searching an index is much faster, but possible more memory comsuming.
searching involves pre-indexing a bunch of data ito a very fast binary
tree.  Only the portions of the index that are being searched are
activated in memory, and unused portions are deactivated.  This is why
mass-indexing makes the Zope process grow so large.  Because almost the
entire index is activated to index everything in your site, the whole
thing is in memory.  incrimental indexing is much more efficient because
it typically only wakes up a portion of the index.

We are working on more and more optimizations for indexing, including
pre-sorting the indexed information in tmp files, so that various
portions of the index can be woken up and put to sleep in an orderly
fasion, instead of randomly as it is now (requiring the consumption of
large amounts of memory).

-Michel

> 
> I'm sure this is quite confusing.  Sorry.  I'm trying to see what the
> best server tuning would be for Zope.  Thanks for your patience.
> 
> All my best,
> 
> Jason Spisak
> webmaster@mtear.com
> 
> _______________________________________________
> Zope-Dev maillist  -  Zope-Dev@zope.org
> http://www.zope.org/mailman/listinfo/zope-dev
> 
> (For non-developer, user-level issues, use the companion list,
> zope@zope.org, http://www.zope.org/mailman/listinfo/zope )
>