[Zope] Zope 2.7.3 Memory Leaks

[Andy Yates]

Still trying to track down an apparent memory leak that started after
upgrading to 2.7.3.

> -----Original Message-----
> From: zope-bounces at zope.org [mailto:zope-bounces at zope.org] On Behalf
Of Andy Yates
> Sent: Thursday, December 02, 2004 4:24 PM
> To: zope at zope.org
> Subject: [Zope] Zope 2.7.3 Memory Leaks
> 
> We recently upgraded from:
> Zope Version (Zope 2.6.1 (source release, python 2.1, linux2), python
> 2.2.2, linux2)
> Python Version 2.2.2 (#1, Feb 24 2003, 19:13:11) [GCC 3.2.2 20030222
> (Red Hat Linux 3.2.2-4)]
> 
> To:
> Zope Version (Zope 2.7.3-0, python 2.3.4, linux2)
> Python Version 2.3.4 (#1, Sep 14 2004, 10:22:31) [GCC 3.2.2 20030222
> (Red Hat Linux 3.2.2-5)]
> 
> Now it seems to leak memory (400+ Mb) to the point I have to restart
> Zope every 3 days.
> 
If Zope is not restarted python2.3 will consume all physical memory and
then start using swap.  The server slows to a crawl and Zope will
eventually stop serving pages.

I've been watching the debug page in the control panel and the refcounts
fluctuate up and down by small amounts but the trend is always up.  They
never seem to level off, they always climb.

Is this typical or does this suggest a problem?

I wrote a new test script in
/usr/lib/Zope-2.7/lib/python/Products/Transience/tests that creates
sessions with large arrays of random data.  Running this test did not
seem to cause any leaks.  However, if I uses sessions in Zope, Zope
seems to leak.  The larger my session data the faster it leaks.

Here is the output from the debug control panel.

# Zope version: (Zope 2.7.3-0, python 2.3.4, linux2)
# Python version: 2.3.4 (#1, Sep 14 2004, 10:22:31) [GCC 3.2.2 20030222
(Red Hat Linux 3.2.2-5)]
# System Platform: linux2
# SOFTWARE_HOME: /usr/lib/Zope-2.7/lib/python
# INSTANCE_HOME: /var/zope
# CLIENT_HOME: /var/zope/var
# Process ID: 22575 (1326767408)
# Running for: 16 hours 36 min 56 sec
# sys.path:
  /var/zope/lib/python
  /usr/lib/Zope-2.7/lib/python/Zope/Startup
  /usr/lib/Zope-2.7/lib/python
  /usr/lib/python23.zip
  /usr/lib/python2.3
  /usr/lib/python2.3/plat-linux2
  /usr/lib/python2.3/lib-tk
  /usr/lib/python2.3/lib-dynload
  /usr/lib/python2.3/site-packages

Class 	January 2, 2005 5:57 pm 	January 3, 2005 8:36 am
Delta
Products.Transience.TransientObject.TransientObject 	4041 	14891
+10850
BTrees.OOBTree.OOBTree 	2019 	11997 	+9978
BTrees.OOBTree.OOBucket 	1297 	5512 	+4215
Products.PageTemplates.Expressions.SubPathExpr 	6669 	9911 	+3242
Products.PageTemplates.Expressions.PathExpr 	2644 	3858 	+1214
Products.PageTemplates.ZopePageTemplate.ZopePageTemplate 	1542
2027 	+485
Shared.DC.Scripts.Bindings.NameAssignments 	880 	1333 	+453
Products.PageTemplates.ZRPythonExpr.PythonExpr 	1062 	1411 	+349
Products.PythonScripts.PythonScript.PythonScript 	679 	833
+154
Products.PageTemplates.Expressions.StringExpr 	294 	448 	+154
OFS.Image.Image 	488 	641 	+153
BTrees.IOBTree.IOBucket 	74 	211 	+137
Shared.DC.Scripts.Signature.FuncCode 	375 	490 	+115
RestrictedPython.PrintCollector.PrintCollector 	188 	303 	+115
Products.ZSQLMethods.SQL.SQL 	519 	629 	+110
Shared.DC.ZRDB.DA.SQL 	117 	198 	+81
Shared.DC.ZRDB.Aqueduct.Args 	117 	198 	+81
App.Extensions.NoBrains 	114 	194 	+80
Shared.DC.ZRDB.sqlvar.SQLVar 	141 	217 	+76
DocumentTemplate.DT_HTML.HTML 	111 	177 	+66
OFS.Folder.Folder 	547 	604 	+57
Products.PageTemplates.Expressions.NotExpr 	200 	254 	+54
Persistence.PersistentMapping 	106 	156 	+50
Shared.DC.ZRDB.sqltest.SQLTest 	85 	134 	+49
OFS.Image.File 	32 	72 	+40
ZPublisher.HTTPRequest.record 	15 	43 	+28
OFS.Image.Pdata 	18 	44 	+26
Products.XMLRPCMethod.XMLRPCMethod.XMLRPCMethod 	13 	37
+24
Shared.DC.ZRDB.sqlgroup.SQLGroup 	35 	50 	+15
DocumentTemplate.DT_Util.Eval 	341 	353 	+12
OFS.DTMLDocument.DTMLDocument 	55 	67 	+12
Products.PlacelessTranslationService.GettextMessageCatalog.GettextMessag
eCatalog 	25 	35 	+10
App.Extensions.FuncCode 	24 	34 	+10
AccessControl.User.User 	11 	20 	+9
App.ApplicationManager.ApplicationManager 	22 	30 	+8
Products.ZMySQLDA.DA.Connection 	20 	28 	+8
Products.SiteErrorLog.SiteErrorLog.SiteErrorLog 	32 	37
+5
AccessControl.User.UserFolder 	31 	35 	+4
DocumentTemplate.DT_Var.Comment 	10 	14 	+4
Products.MailHost.MailHost.MailHost 	8 	12 	+4
App.Product.ProductFolder 	10 	13 	+3
Products.PlacelessTranslationService.PlacelessTranslationService.Placele
ssTranslationService 	9 	12 	+3
App.ApplicationManager.DebugManager 	7 	9 	+2
pyrad.dictionary.Dictionary 	4 	6 	+2
pyrad.client.Client 	4 	6 	+2
DateTime.DateTime.DateTime 	71 	72 	+1
ZServer.medusa.counter.counter 	31 	32 	+1
Products.ZMySQLDA.db.DB 	19 	20 	+1
ZServer.HTTPServer.zhttp_channel 	6 	7 	+1
asynchat.fifo 	5 	6 	+1 

Thanks!
Andy