[Zope-CMF] Re: Bulk Indexing

[Tres Seaver]

Shane Landrum wrote:
> On Mon, Jan 26, 2004 at 09:09:51AM +0000, Chris Withers wrote:
> 
>>Shane Landrum wrote:
>>
>>
>>>How many ZEO clients do you have running total when you do 
>>>this? 
>>
>>Only one that does and content writing ;-)
>>
>>
>>>Do you have trouble with database ConflictErrors?  
>>
>>...as a result, no, not really...
> 
> 
> Ah, right. That's not possible in my environment.
> 
> 
>>>It's quite possible that I'm dealing with a unique situation here.
>>>We have 7 large Zope servers talking to a very large ZEO server,
>>>and unusually for lots of Zope installs, we do a lot of database
>>>writes, more or less all the time. We have an elaborate automatic
>>>news feed system that's always dumping new content into the database,
>>>an automatic workflow approval system for some of that new content, 
>>>and a staff of 2-15 editorial people on the system at any one 
>>>time, poking and prodding at content.  As a result, it's 
>>>relatively likely that at least one object out of several hundred 
>>>thousand is going to be being worked on when the reindexing is 
>>>done; hence, ConflictErrors. 
>>
>>Have you tried the QueuedCatalog?
> 
> 
> No, and it doesn't come up on the zope.org products search. However,
> a Google search and some poking around solved the problem for me.
> 
> For the benefit of the archives:
> http://cvs.zope.org/Products/QueueCatalog/ is the CVS version
> of QueuedCatalog. From the docstring:
> 
> --------------
>  A QueueCatalog delegates most requests to a ZCatalog that is named
>  as part of the QueueCatalog configuration.
> 
>  Requests to catalog or uncatalog objects are queued. They must be
>  processed by a separate process (or thread). The queuing provides
>  benefits:
> 
>  - Content-management operations, performed by humans, complete
>  much faster, this making the content-management system more
>  effiecient for it's users.
> 
>  - Catalog updates are batched, which makes indexing much more
>  efficient.
> 
>  - Indexing is performed by a single thread, allowing more
>  effecient catalog document generation and avoiding conflict
>  errors from occuring during indexing.
> 
>  - When used with ZEO, indexing might e performed on the same
>  machine as the storage server, making updates faster.
> --------------
> 
> Looks potentially useful. Thanks Chris.

Shane,

Chat with Marc about this one:  we already have QueueCatalog in the 
buildout, but never wired it in, largely because it has expectations of 
the "real" catalog which the asset repository doesn't fulfill.

Tres.
-- 
===============================================================
Tres Seaver                                tseaver at zope.com
Zope Corporation      "Zope Dealers"       http://www.zope.com