[Zope-dev] Specialist/Rack scalability

[Phillip J. Eby]

At 07:12 PM 1/21/01 +0000, Steve Alexander wrote:
>
>So, storing things in a Rack happens in a number of stages:
>
>   Your application interacts with the Rack
>   The Rack (perhaps) stores the object persistently in its BTree
>   The BTree is a collection of persistent ZODB objects
>   The ZODB objects are stored as Python Pickles in a FileStorage
>
>We can consider what the effect of storing 60 000 objects is at each of 
>these interfaces.
>
>The Rack shouldn't have a problem with 60 000 objects.
>
>I doubt a BTree would have a problem.
>
>The ZODB might not like accessing many large objects during a single 
>transaction, as all those objects need to be in memory at once.
>
>A FileStorage should have no problem reading 60 000 stored objects. 
>However, if these objects are changing much, your Data.fs will grow 
>fast. In any case, you may find undo and history screens take a long 
>time to appear.
>
>However, if you are using a Rack, you have a lot of choice about where 
>you put your data. You can put frequently changed aspects of your data 
>on the filesystem, and the rest in FileStorage for example.

Just to expand a little on the abov...  Racks should scale at least as
well, if not larger than a ZCatalog, given the same storage backing for the
ZODB.  This is because ZCatalog has to manage a minimum of one forward and
reverse BTree for *each* index, plus another few BTrees for overall storage
and housekeeping.  Also, keyword and full text indexes store multiple BTree
entries per object, so that's a factor as well.

So don't worry about the Rack.  If you're using a Rack, you can store the
data anywhere, and you can index it in an RDBMS, LDAP directory, ZCatalog,
or some combination thereof, using triggers to keep the data in sync.