[Zope] LocalFS and mega-uploads

[larry_prikockis@NatureServe.org]

...Just to kind of wrap this thing up... Thanks to those who helped me
figure this out.  As several people have now suggested and confirmed, the
problem was the partition I had /var/tmp on filling up during the upload
process.

Moving /var/tmp to a bigger partition cured the basic problem I was having.
However, transferring of extremely huge files still put more of a load on
the system than I would've liked.  In the end, what I've ended up doing is
setting up a separate standalone FTP server, locking it down as much as
possible and using an html/script front-end to allow some degree of
integration with the rest of my Zope site.

It's not quite as nicely integrated as using Zserver's native http/ftp
capabilities, but it does the job.

> -----Original Message-----
> From: Jim Penny [mailto:jpenny@universal-fasteners.com]
> Sent: Wednesday, January 16, 2002 7:22 PM
> To: larry_prikockis@NatureServe.org
> Subject: Re: [Zope] LocalFS and mega-uploads
> 
> The /tmp path hypothesis does seem correct.  I just started a 
> megaupload, 
> and found this--that space went down as the upload 
> progressed, (as reported
> by df -k).  But there were no unusual files in /tmp!
> 
> However, looking at lsof's output, I see:
>                                                  bytes
> 						 vvvv
> python1.5 32012     zope   24u   REG        3,1 56111104    448341
> /tmp/@31823.224 (deleted)
> 
> So, it looks like cgi.py is being activated, and that it is stuffing
> the upload into a file that it is immediately unlinking.  The upshot 
> of this is that the file is not present to prying eyes.  My script
> was then reading it from /tmp in 4k chunks.
> 
> This being the case, the best thing to do is to make sure that /tmp
> has a LOT of room available.  You want to make sure that it is mounted
> on a separate partition from the data's destination and preferably
> from anything else.  Otherwise, your hard drive will have to have
> at least twice as much space free (one for the /tmp copy and one for
> the destination copy) as the largest expected upload.
> 
> Note, it is fairly conventional to have /tmp mounted on /.  Moreover,
> many administrators keep / intentionally small.  You don't want to 
> do this here:  you can keep / small, but only if /tmp is mounted
> elsewhere!