[Zope] ZEO with Multiple Storages

[Jonathan]

----- Original Message ----- 
From: "Brian Brinegar" <brinegar at ecn.purdue.edu>
To: <zope at zope.org>
Sent: Wednesday, September 13, 2006 9:22 PM
Subject: [Zope] ZEO with Multiple Storages


> We currently have a ZEO environment with FileStorage (data.fs) over 100 
> gigs, which continues to grow. Currently we are approaching a major 
> hardware and software upgrade. All new hardware and moving from Zope 2.6 
> to Zope 2.9.
>
> Our server hosts sites for several different schools and departments 
> within our college, what I have considered doing is trying to split up the 
> data.fs into multiple smaller storages, possibly one per school. I suspect 
> if this is possible it could potentially have several benefits. The first 
> being that we would not have to deal with 100 gig files which are very 
> difficult to move around backup/restore, etc. Secondly, we often have a 
> very large number of users updating content at a given time and will run 
> into slowdowns where a transaction is blocked while another commits. I 
> hope that multiple storages could allow for a transaction to commit to one 
> storage at the same time as another, though I am not sure. Along this line 
> can a single transaction commit to multiple storages?

If you mean having zope commit a single transaction to multiple storages, 
then 'vanilla' zope won't do it (Zope Corp has a commercial product called 
ZRS which does this type of thing).

Alternatively, if your server is disk-bound then you could look into RAID 
disk striping to try to improve performance.


> First off, does this make any sense? Is it worth pursuing? Secondly, I 
> assume that where an object is stored does not limit what other objects it 
> can access in a transaction (e.g. object A could add a property to object 
> B if they were in different storages) because all changes would occur in a 
> client. (this ties into the single transaction committing to multiple 
> storages)
>
> If this does makes sense is there any documentation, recommendations, best 
> practices for how to set something like this up?
>
> Lastly, any thoughts on how we could split up our existing data.fs file? 
> One thought I had was to attempt to import/export data.


If you have the time and hardware you could easily set up several zeo 
servers on different 'back-end' hardware servers (ie. put each zeo server on 
its on computer) to improve performance, increase reliability/robustness, 
etc.  Your 'use case' in which you have different schools and departments 
would lend itself to this kind of distribution (put the biggest/heaviest 
users on their own servers, group together smaller/lighter users on shared 
servers).

You just need to define a mount point (in zope.conf) for each different zeo 
server (and set up each zeo server of course).

If you already have the data for the various schools & depts stored in 
separate folders, it would be easy to export each 'school' folder and then 
import it into the new configuration.


hth

Jonathan