[Zope-DB] Large jobs and timeouts.

[Michal Kurowski]

Hi,

I'd like to ask for your advice on a problem with ZSQL methods
timeouts.

I get a timeout when I try to perform *really* large jobs with some
inserting methods.

I have five rather big tables and a python script wrapper that does
the following:

1) fires up a "select" method on those tables to find out if a column
is emty or not
2) calls an external method (quite time consuming) to do some
computetion (file system consulting as well unfortuantelly)
3) fetches results of those computations and calls another sql method
to insert them into a column

Something like this:

for i in tables:
    tab = []
    for results in container.empty_ss(table=i):
        if results.pdb != '':
            tab.append((results.internal_id, results.pdb,
	    results.seq))
        out[i] = tab

	for x in  out.items():
	    for	y in range(0, len(x[1])):

	    table_id = x[1][y][0]

	    table_pdb = string.split(x[1][y][1], '_')
	    if len(table_pdb) == 2:
	       table_chain = table_pdb[1]
	       table_pdb = table_pdb[0]
	    elif len(table_pdb) == 1:
	       table_chain = 'single'
	       table_pdb = table_pdb[0]

	    table_seq = x[1][y][2]

	    ss = container.external_update(table_seq, table_pdb, table_chain)
	
	    container.insert_ss(table='i', table_id=table_id, ss=ss)

That works pretty well but only for several rows...


What I need to do is to make sure it will complete the job and fill
all the rows that are empty.
And I'd rather not make it fully external (with cron instead of Xron).

Is that possible ?


-- 
Michal Kurowski
<mkur@poczta.gazeta.pl>