[Zope] speeding up multiple file upload

zope at netchan.cotse.net zope at netchan.cotse.net
Tue Mar 9 12:26:41 EST 2004


Our just finished project implements a product which includes a domain
specific repository-like thing what have to handle multiple (100-1000)
file uploads. In a new project, we will implement a more general version
of this repository as a standalone product. The old one works quite well
but there's a lot of room for improving performance in which I seek your
advices now.
The relevant part of the old stuff is a client-server application. The
following steps are made in a transaction:
- client generates a filelist, sends it to the server [xml-rpc]
- server determines which files should be uploaded (changed files) sends
back the result [xml-rpc]
- client collects all the files into a large zip (they are more or less
regular web content, well compressible), sends to the server [http
multipart/form-data]
- server unzips and stores the files in ZODB 

All communication is done via ssl, so the connect-disconnect operation
is quite expensive. The two longest operation is the upload and the
unzip/store and theoretically these could be done in an asynchronous
consumer-producer setup. I'm looking for something like network copy of
multiple files via tar, e.g.: 
tar -xzf- | ssh remoteserver "tar -czf-"
The aim is to send all the files over the same channel, one by one, and
start server side processing as soon as the first file arrived. Now,
this is completely doable with pure python, the question is how should I
do this when Zope is the receiver? Both the client and the server part
is under our control so any wild idea would work, the only restriction
is the all communtication must go over port 443, because both the
clients and the server are firewalled.
Any hints or pointers would be appreciated.

Regards,
Sandor




More information about the Zope mailing list