[Zope-dev] Non-undoable storage

[Chris McDonough]

Shane Hathaway wrote:
> 
> "Morten W. Petersen" wrote:
> > during testing of a mail product I've discovered that the Data.fs file may
> > bloat considerably after storing 50 messages.  Packing the database will
> > reduce the Data.fs file to 20 MB (from 40 MB).  Another thing is that
> > storing 50 messages takes a *long time* on a 600Mhz 256 MB RAM system.
> 
> Did you catalog each message?  What version of Zope?

Hold your tongue, Shane!  ;-)
 
> > The thing that struck me was that transactions may cause a lot of
> > overhead, both in database size and for execution speed, so, the thing I'm
> > wondering about is if there exists a non-undoable storage, or something
> > similar that doesn't support transactions..
> 
> Yep, see BerkeleyStorage.
> 
> > Another thing I noticed BTW, is that retrieving 500 messages and packing
> > the database on every 50th message (all messages retrieved during one
> > request) would lead to off the charts memory usage, the process ended up
> > using 700 MB of RAM (had to add swap files as the footprint grew!)
> 
> You could either:
> 
> 1) Import messages in a single transaction with subtransactions.  This
> would take less time and incur less memory consumption and database
> bloat.  We at DC have experimented with bulk-loading mail into Zope and
> this is what works the best.

See ZCatalog's ZCatalog.py/catalog_object method for an example of using
subtransactions.  Note that the cacheFullSweep call in there is
necessary to see any RAM savings.

> 
> 2) Change the database cache parameters to keep the RAM cache as low as
> possible (set the timeout to 3 seconds and the target size to 10).
> 
> 3) Manually zap the caches periodically, which is a capability of Zope
> 2.4.x.



> 
> Shane
> 
> _______________________________________________
> Zope-Dev maillist  -  Zope-Dev@zope.org
> http://lists.zope.org/mailman/listinfo/zope-dev
> **  No cross posts or HTML encoding!  **
> (Related lists -
>  http://lists.zope.org/mailman/listinfo/zope-announce
>  http://lists.zope.org/mailman/listinfo/zope )