[Zope] Why can't I add lots of objects at once?

Kyler B. Laird laird@ecn.purdue.edu
Wed, 11 Apr 2001 12:42:48 -0500


A month or two ago I asked here for advice on putting
roster data (~233K entries/semester) in our database.
One of the suggestions was to use XML-RPC in order to
avoid the timeouts I was getting.

I've been working on an XML-RPC solution and it is
quite adequate for small data sets, but if I try to
add a few hundred student/class entires, it hangs.

After a few iterations, I've come up with this script
on the Zope side.  Essentially, it creates a
structure like
	AAE
		203
			01
				01
					students
						kyler.b.laird.1
						sally.r.smith.5

===================================================	
		if (not csubject in courses.objectIds()):
			courses.manage_addFolder(csubject, csubject)
		if (not cnumber in courses[csubject].objectIds()):
			courses[csubject].manage_addFolder(cnumber, cnumber)
		if (not division in courses[csubject][cnumber].objectIds()):
			courses[csubject][cnumber].manage_addFolder(division, division)
		if (not section in courses[csubject][cnumber][division].objectIds()):
			courses[csubject][cnumber][division].manage_addFolder(section, section)

		if (not 'students' in courses[csubject][cnumber][division][section].objectIds()):
			courses[csubject][cnumber][division][section].manage_addFolder('students', 'students')


		current_students = courses[csubject][cnumber][division][section]['students'].objectIds()

		for student in students:
			if (not student in current_students):
				courses[csubject][cnumber][division][section]['students'].manage_addFolder(student, student)
===================================================	


Terribly naive programming aside, what's wrong with
calling this a couple hundred times?  It seems to
just keep taking more and more of the processor
time.  Eventually I give up, kill the server and 
restart it.  Then I can run the script again and it
will get a little bit further (because it doesn't
have to create the existing objects, I assume).

Of course, I'd prefer not to have to break this up
into one thousand chunks with server restarts in
between each one.

Any suggestions?

Thank you.

--kyler