[Zope] Force a ZODB transaction commit?

[Roché Compaan]

* Jim Kutter <jim at ebizq.net> [2004-07-22 00:01]:
> Hi
> 
> I'm trying to use ZCatalogs excellent indexing capability as a front end
> for data mining. We have 80k records which I'm reading using a ZSQL
> method (which also uses a custom base class for the records), for each
> record I then have to pull in data from three other tables, then I add
> that virtual object to the catalog. All of this is done via a python
> script.
> 
> Here's the kicker:
> 
> Zope takes forever to do this, and it tends to freeze the zope instance
> while it's doing it, in fact I have yet to successfully do this for a
> batch size larger than 100 records.

I am sure that the catalog is not the culprit here - I regularly index
300k objects in one go. Your instance freezing could be due to memory
consumption, committing a transaction every n records will help keep
that in check.

> My suspicion is that zope is doing this all in a transaction, and not
> committing the change per object, but instead waiting for the whole 80k
> records to be worked on, then commit the whole bag. Needless to say this
> isn't optimal. Is there a way I can force the transaction to commit
> after each record is done?

get_transation().commit()

but it won't work in a Python Script, use it in an external method.

-- 
Roché Compaan
Upfront Systems                 http://www.upfrontsystems.co.za