[Zope] why can zope not handle large files

Eric Balasbas ebalasba@bway.net
Fri, 18 May 2001 18:24:07 -0400 (EDT)


Sounds like the problem is with inserting many rows into the database in
one transaction. Breaking up the inserts into many transactions should
help.
How exactly do you put the csv file into Zope?

Eric Balasbas
Senior Developer
eric@virtosi.com

http://www.virtosi.com/
Virtosi Ltd.
Design -- Branding -- Zope

On Fri, 18 May 2001, Christopher N. Deckard wrote:

> For a project we are working on we have export of an MS Access
> database as csv (comma seperated) files.  The csv file is about 37MB
> with rows of 18 numbers sepearted by commas.  The file has been
> uploaded into Zope and then we are trying to insert each row into a
> PostgreSQL database using either the PoPy or PsycoPG database
> adapter.  Basically Zope never comes back from trying to do this.
> We have also tested against Oracle with the same results.  It's been
> run on an Athlon 600 and a Sun E450.  Zope just starts to take up
> more and more memory eventually filling up swap and never completing
> the transaction.  If we split the 37MB file up into small 3-4MB
> chunks, everything works fine.  But it just can't handle the larger
> file.  To me, 37MB doesn't seem like much.  If it were .5GB, I could
> understand Zope having problems.  How can I get this to work?
>
> I am not on the zope list (though I'll probably subscribe here in a
> moment) so please CC me on a reply.
>
> Thanks,
> -Chris
>
> _______________________________________________
> Zope maillist  -  Zope@zope.org
> http://lists.zope.org/mailman/listinfo/zope
> **   No cross posts or HTML encoding!  **
> (Related lists -
>  http://lists.zope.org/mailman/listinfo/zope-announce
>  http://lists.zope.org/mailman/listinfo/zope-dev )
>