[ZWeb] Zope.org out of space again

Jim Fulton jim@digicool.com
Mon, 11 Dec 2000 10:48:55 -0500


Martijn Pieters wrote:
> 
> On Mon, Dec 11, 2000 at 10:18:35AM +0100, Martijn Pieters wrote:
> > I got a 'No space left on device' and started a pack.
> 
> Then again, I triggered the error when I tried to update the SiteIndex
> Catalog, and it happened in TmpStore.py:
> 
> Traceback (innermost last):
>   File /usr/local/zope_soft/lib/python/ZPublisher/Publish.py, line 222, in publish_module
>   File /usr/local/zope_soft/lib/python/ZPublisher/Publish.py, line 187, in publish
>   File /usr/local/zope_soft/lib/python/Zope/__init__.py, line 221, in zpublisher_exception_hook
>     (Object: Traversable)
>   File /usr/local/zope_soft/lib/python/ZPublisher/Publish.py, line 171, in publish
>   File /usr/local/zope_soft/lib/python/ZPublisher/mapply.py, line 160, in mapply
>     (Object: manage_catalogReindex)
>   File /usr/local/zope_soft/lib/python/ZPublisher/Publish.py, line 112, in call_object
>     (Object: manage_catalogReindex)
>   File /usr/local/zope_soft/lib/python/Products/ZCatalog/ZCatalog.py, line 290, in manage_catalogReindex
>     (Object: Traversable)
>   File /usr/local/zope_soft/lib/python/Products/ZCatalog/ZCatalog.py, line 383, in catalog_object
>     (Object: Traversable)
>   File /usr/local/zope_soft/lib/python/ZODB/Transaction.py, line 251, in commit
>   File /usr/local/zope_soft/lib/python/ZODB/Connection.py, line 351, in commit
>     (Info: (('IIBTree', 'Bucket'), '\000\000\000\000\000\031\\S', ''))
>   File /usr/local/zope_soft/lib/python/ZODB/TmpStore.py, line 142, in store
> IOError: (see above)

This was the temporary storage used by subtransactions.
 
> So maybe it *was* the device that ran out of space or the file that
> crossed the 2GB line. But I don't have access to the ZEO server to verify
> this.

The temporary storage is on the ZEO clients, not on the server.
Subtransactions are done using s special storage. The normal storage
(ZEO ClientStorage) doesn't see any data till the outer transaction is
committed.

> Maybe it is time we moved to Shane's new PartionedStorage? Or did you
> already have plans?

This has nothing to do with the temporary storage.

The clients need to have lots of temporary space.
If this temporary space isn't in the usual location, then
you need to set the TMPDIR environment variable to 
tell Zope (actually, yje tempfile module) where the
space is.  This is what was done on the old
zope.org site.

Note that you could, concievably, dedicate one
client to do this sort of thing, or even 
write a ZEO-Client python script that does just
the catalog reindexing in a dedicated short-lived 
process. This is actually probably the right way
to proceed.

Jim

--
Jim Fulton           mailto:jim@digicool.com   Python Powered!        
Technical Director   (888) 344-4332            http://www.python.org  
Digital Creations    http://www.digicool.com   http://www.zope.org