[Zope] Zope 2.7 FTP Bandwidth Limiter?
Chris McDonough
chrism at plope.com
Sun Mar 28 06:18:49 EST 2004
FWIW, I think I tracked this down to a too-small buffer size in the FTP
server code. The latest code from CVS (Zope-2_7-branch) allows for much
faster download speeds (~ 100X), at least for me.
On Fri, 2004-02-27 at 11:59, Edward Pollard wrote:
> The profiler seems to offer absolutely no information of use. I suspect
> it does not include Medusa performance.
>
> The only thing that popped up was a call to FileStorage.py, but that
> call did not seem to monopolize the CPU.
>
> Attached are three logs
> 1) The FTP session
> 2) The profiler data for that time period.
> 3) the TOP data for that period
>
>
> Due to the bugs in 2.6.2, I've had to disable FTP access to the server,
> which has my clients infuriated. Any help at all with this would be
> most grateful. The only other option is to got back to 2.6.0, but that
> means we can't compress the ZODB without truncating it to 4 bytes.
>
> Ed
>
> 230 Login successful.
> Remote system type is UNIX.
> Using binary mode to transfer files.
> ftp> put big_file.exe
> local: big_file.exe remote: big_file.exe
> 227 Entering Passive Mode (127,0,0,1,202,169)
> 150 Opening Binary connection for big_file.exe
> 226 Transfer complete.
> 18817367 bytes sent in 0.288 secs (6.4e+04 Kbytes/sec)
> ftp> get large_bin.exe
> local: large_bin.exe remote: large_bin.exe
> 227 Entering Passive Mode (127,0,0,1,202,156)
> 150 Opening Binary mode data connection for file 'large_bin.exe'
> ########################################################################
> ########################################################################
> ########################################################################
> ########################################################################
> ########################################################################
> ########################################################################
> receive aborted
> waiting for remote to finish abort
> 426 Connection closed; transfer aborted
> 226 ABOR command successful.
> 442368 bytes received in 35.2 secs (12 Kbytes/sec)
> ftp> exit
> 221 Goodbye.
>
>
> Here is the Profiler data that covers the timeframe of this activity
> (where tottime > 0)
>
> Ordered by: internal time
> List reduced from 446 to 100 due to restriction <100>
>
> ncalls tottime percall cumtime percall filename:lineno(function)
> 3 0.280 0.093 0.280 0.093
> ApplicationManager.py:183(refcount)
> 21 0.040 0.002 0.050 0.002
> Connection.py:598(_set_ghost_state)
> 812 0.040 0.000 0.040 0.000 DT_HTML.py:23(search)
> 21 0.040 0.002 0.040 0.002 FileStorage.py:651(_load)
> 1007 0.020 0.000 0.020 0.000
> HTTPRequest.py:1211(__getitem__)
> 16 0.020 0.001 0.310 0.019 DT_In.py:618(renderwob)
> 114 0.020 0.000 0.040 0.000 HTTPRequest.py:1231(keys)
> 473/197 0.020 0.000 0.020 0.000
> DT_Util.py:341(parse_params)
> 22 0.020 0.001 0.030 0.001
> PersistentExtra.py:23(bobobase_modification_time)
> 96/39 0.020 0.000 0.050 0.001
> DT_String.py:244(parse_close)
> 117 0.020 0.000 0.040 0.000 DT_Var.py:170(__init__)
> 1638 0.020 0.000 0.020 0.000 DT_Var.py:183(<lambda>)
> 725 0.020 0.000 0.030 0.000 DT_String.py:95(_parseTag)
> 181 0.010 0.000 0.010 0.000 DT_Util.py:229(name_param)
> 24 0.010 0.000 0.010 0.000
> DateTime.py:352(_julianday)
> 2 0.010 0.005 0.010 0.005
> ApplicationManager.py:285(version_txt)
> 134 0.010 0.000 0.010 0.000 DT_String.py:186(skip_eol)
> 19/17 0.010 0.001 0.010 0.001
> Traversable.py:95(getPhysicalPath)
> 249 0.010 0.000 0.010 0.000
> Connection.py:180(_persistent_load)
> 77/75 0.010 0.000 0.120 0.002 DT_Util.py:175(eval)
> 725 0.010 0.000 0.010 0.000 DT_HTML.py:129(start)
> 1158 0.010 0.000 0.010 0.000 HTTPRequest.py:1090(get)
> 7 0.010 0.001 0.010 0.001
> HTTPRequest.py:148(setServerURL)
> 8 0.010 0.001 0.010 0.001 posixpath.py:171(exists)
> 6 0.010 0.002 0.010 0.002
> sre_compile.py:229(_mk_bitmap)
> 725 0.010 0.000 0.010 0.000 DT_HTML.py:152(parseTag)
> 11 0.010 0.001 0.020 0.002
> VirtualHostMonster.py:130(__call__)
> 8 0.010 0.001 0.020 0.003 DT_String.py:514(read_raw)
> 59/20 0.010 0.000 0.180 0.009
> DT_String.py:195(parse_block)
> 2 0.010 0.005 0.200 0.100
> ApplicationManager.py:204(refdict)
> 78 0.010 0.000 0.010 0.000 User.py:171(allowed)
> 11 0.010 0.001 0.010 0.001
> HTTPResponse.py:266(setBody)
>
>
> TOP data, where %CPU > 0:
>
> PID USER PRI NI SIZE RSS SHARE STAT %CPU %MEM TIME CPU
> COMMAND
> 15732 apache 25 0 80292 78M 3624 R 99.8 7.7 1:25 0
> python2.3
> 15807 root 15 0 1252 1252 1016 S 0.2 0.1 0:00 0 ftp
> 1 root 15 0 476 448 424 S 0.0 0.0 0:04 0 init
>
>
> On Feb 26, 2004, at 10:40 AM, Chris McDonough wrote:
>
> > The 100% CPU spike is likely the cause of the slow performance. I
> > think
> > a reasonable strategy might be to turn on profiling in Zope 2.7 (see
> > zope.conf profile-publisher? key, and Control_Panel -> Debug) to see
> > what falls out of that. I'm sure there's a lot of information about
> > this if you google for "zope profiling".
> >
> > On Thu, 2004-02-26 at 08:26, Bobb wrote:
> >> Which OS / processor / memory are you running Edward ? Which network
> >> type ?
> >> Ethernet/speed, etc.
> >> Does the machine have an non-zope ftp server on it?
> >> If you use that do you get the same results?
> >> I find the 100% cpu spike even stranger than slow performance... (but
> >> that's
> >> just me ;) )
> >>
> >> ----- Original Message -----
> >> From: "Edward Pollard" <pollej at uleth.ca>
> >> To: "Bobb" <rawbobb at hotmail.com>
> >> Cc: <zope at zope.org>
> >> Sent: Wednesday, February 25, 2004 3:21 PM
> >> Subject: Re: [Zope] Zope 2.7 FTP Bandwidth Limiter?
> >>
> >>
> >> Indeed we still only get 10k/sec via ftp to localhost.
> >>
> >> It is worth noting that FTP upload is full-speed. Also noteworthy is
> >> that the CPU on this box is going to 100% when I download via FTP (45%
> >> user/55% system).
> >>
> >> This problem is exhibited, pretty much identically, in Zope 2.6.4 and
> >> 2.7.
> >>
> >> Zope 2.6.2 seems to have critical bugs in its FTP implementation,
> >> which
> >> means I needed to be off of it yesterday.
> >>
> >> • Dieter does not have this problem in 2.7, which may suggest that the
> >> problem is not Zope.
> >>
> >> • We are running different versions of Python (2.1 for Zope 2.6.4,
> >> 2.3.3 for Zope 2.7) for each of these servers, which may suggests it
> >> isn't a Python-related problem, unless there is something we did wrong
> >> twice. (And our 2.6.2 is running fine on Python 2.1, although it is a
> >> separate install of it than the one 2.6.4 uses).
> >>
> >> • Everything else runs at a proper transfer rate on this machine - and
> >> executing the 2.6.4 and 2.7 installs on the machine where we run 2.6.2
> >> as a process successfully yields the same results - which may suggest
> >> that the hardware is fine.
> >>
> >> Conclusion: I got nothing.
> >>
> >> Does anyone have any thoughts at all on this subject?
> >>
> >> Ed
> >>
> >>
> >> On Feb 22, 2004, at 11:51 AM, Bobb wrote:
> >>
> >>> I'm curious what about ftp'ing locally 127.0.0.1:ftpport ?
> >>> still 10k/sec?
> >>>
> >>> ----- Original Message -----
> >>> From: "Edward Pollard" <pollej at uleth.ca>
> >>> To: <zope at zope.org>
> >>> Sent: Saturday, February 21, 2004 4:20 PM
> >>> Subject: [Zope] Zope 2.7 FTP Bandwidth Limiter?
> >>>
> >>>
> >>>> We're experimenting with Zope 2.7, and are currently unable to get
> >>>> FTP
> >>>> transfers to go any faster that 10k/sec. This seems to be some sort
> >>>> of
> >>>> arbitrary limit, as we have plenty of bandwidth and horsepower
> >>>> available. Indeed, we can pretty much make as many FTP sessions as
> >>>> we
> >>>> want simultaneously, they just all go at 10k/sec.
> >>>>
> >>>> Was there some sort of limiter or throttle introduced somewhere
> >>>> along
> >>>> the lines that I am unaware of?
> >>>>
> >>>> ---
> >>>> Edward J. Pollard, B.Sc
> >>>> Webmaster, University of Lethbridge
> >>>>
> >>>>
> >>>> _______________________________________________
> >>>> Zope maillist - Zope at zope.org
> >>>> http://mail.zope.org/mailman/listinfo/zope
> >>>> ** No cross posts or HTML encoding! **
> >>>> (Related lists -
> >>>> http://mail.zope.org/mailman/listinfo/zope-announce
> >>>> http://mail.zope.org/mailman/listinfo/zope-dev )
> >>>>
> >>
> >>
> >> _______________________________________________
> >> Zope maillist - Zope at zope.org
> >> http://mail.zope.org/mailman/listinfo/zope
> >> ** No cross posts or HTML encoding! **
> >> (Related lists -
> >> http://mail.zope.org/mailman/listinfo/zope-announce
> >> http://mail.zope.org/mailman/listinfo/zope-dev )
> >>
> >>
> >> _______________________________________________
> >> Zope maillist - Zope at zope.org
> >> http://mail.zope.org/mailman/listinfo/zope
> >> ** No cross posts or HTML encoding! **
> >> (Related lists -
> >> http://mail.zope.org/mailman/listinfo/zope-announce
> >> http://mail.zope.org/mailman/listinfo/zope-dev )
> >
>
>
> _______________________________________________
> Zope maillist - Zope at zope.org
> http://mail.zope.org/mailman/listinfo/zope
> ** No cross posts or HTML encoding! **
> (Related lists -
> http://mail.zope.org/mailman/listinfo/zope-announce
> http://mail.zope.org/mailman/listinfo/zope-dev )
More information about the Zope
mailing list