[Zope] Network Appliances NetCache vs. Squid

Tue, 6 May 2003 23:45:24 -0700

Well Marc, Paul browning brought up a good point about the Solaris and
threading issues. That could explain a lot. I ran Zope 2.3 on a Celeron
400 Linux box with very high traffic and it held up well. 15 to 20 to 1
minute is nothing I have ever seen a Zope-driven site do. That's not
good!

As far as the images, you would use Zope RAM Cache Manager to store the
accessed images in RAM. If most of your images are around 10k, you might
multiple that by how many images you use on the site to figure out how
much ram it will take up. You should also cache pages that are
pretty-much static. If you use a header and footer or zpt's metal: for
templating where the content gets put inside the design template, you
can cache the template or header and footer too.

Another thing on using squid to cache the images and pages, you can
specify what objects are cacheable and Zope will use ICP to tell Squid
the page or image has changed and Squid will re-cache the object. I am
not sure about the relative directory issue with the images. It might
not matter when using Squid. It would also seem to me that having an
absolute reference to the image would speed things up a bit because Zope
won't have to use acquisition to find the image. Just a thought, I may
be wrong.

I would seriously see if the threading issue and Solaris is the culprit
for your slow responses. Is there any reason why you run Solaris?
Another way I have found to speed-up Zope (this sounds strange but it
works) is to run Zope behind Apache and Apache behind Tux. I have done
that before and it is a very speedy solution.

-----Original Message-----
From: zope-admin@zope.org [mailto:zope-admin@zope.org] On Behalf Of Marc
Burgauer
Sent: Wednesday, April 30, 2003 4:36 AM
To: zope@zope.org
Subject: Re: [Zope] Network Appliances NetCache vs. Squid

Hi Scott, thanks for the reply!

This is going to be a bit lengthy, as I tried to fully describe the
problem
at hand.

Interestingly enough, I started out asking this list how I can determine
best usage of the cache mechanisms built into Zope. There are two
bottlenecks in the site.

1.) Images

We are only responsible for programming and hosting of the site. The
customer has chosen to give the HTML coding part of the work to a design
agency.

All pages, images and data are stored in the ZODB!

We have trained the HTML coders (2 years ago) and written some tip
sheets
explicitly telling them to use absolute referencing for images from the
root-level images-directory.
However in a recent re-design of the site, they have ignored this advice
and
all images are now referenced relatively. In any bog standard web server
environment, those paths wouldn't work. (As they are simply wrong.) But
thanks to Zope inheritance, it just climbs up the directory tree and
ultimately finds the images. So the coders never spotted their mistake.

Example:

The images are all stored in a root-level image directory or
subdirectories
of it:

/images/navigation/home.gif
/images/mainlogo.gif
/images/navigation/news/nextitem.gif

The site contains 45 "microsites" which have identical page-structure,
the y
have all the same pages, but different data. So each microsite has a
news
page. On such a news page all three images above are required. The HTML
code
for these images reads:

The microsites are all root level directories. The news pages are in
subdirectories of each microsite. The templates use "includes" for most
of
the page components and these includes are in the root. For example:

newslisting.dtml
/newyork/news/list
/london/news/list
/paris/news/list
/munich/news/list

A browser or proxy trying to cache things will resolve any of the images
to
be "local" to the news directory and not identify them all being the
same
image. So caching will only work for pages within the same subdirectory,
but
not across all pages on the site, even they all use the same logo.

Obviously, I could go and edit the 100+ include files and fix the false
referencing. However, that would only fix about half of the
image-caching
problem, as some images are stored in image folders within each
microsite.

Obviously, this task is labour intensive and I look for a quicker fix.
Also,
the person responsible for the site within the customer has changed and
the
new guy wants to change the site's look and feel in around 6 months
(when he
has new budget) to be less image rich. At the moment each page contains
at
least 30 images, most being albeit small. But as they are mainly text
links
rendered to images using a not-so-common font (Stone) we (me and the
customer) agree that making the real text links in the next revamp will
improve usability and speed up page load time.

Is Zope's RAM cache manger going to help delivering the images here?
Most of
them are under 10Kb in size. Obviously the times saved would be fetching
them from the ZODB on disk.

2.) Zope responding to request

Zope is responding slowly to any initial request. You can see in the
browser's status bar the "connecting to site" message for about 15-20
seconds before anything happens, at peak usage time this can go up to
nearly
a minute. The site is "busy" from around 6am to 1am, with a peak usage
between 4pm and 7pm.

I believe that the issue here is how many requests can be served in
parallel. The box itself could serve more, neither CPU, RAM nor network
are
maxed out. CPU load is usually around 30% even at peak usage. So I was
wondering if running ZEO and 2 instances of Zope pointing at the same
ZODB
on the same machine would improve things.

Content changes frequently overall, but many pages don't change for
weeks.
Most pages are truly dynamic (i.e. content managed). Packing the
database
every night has not improved performance. And because of the true
dynamic
nature of the site I wonder how well a cache like squid would improve
the
site. (I have never done any caching, although recently read a lot about
this.)

We're using Zope 2.3 on a Solaris 8 server (Compaq, 2 P2 450MHz CPUs,
RAID
Disk array, 1 Gig of RAM). The box and OS has been chosen by the
customer
and there's no room for play currently. (OS is company policy and no
money
for another box.)

We have developed a few "light-weight - very easy to use" content
management
products for Zope and have run into a few problems when testing them
against
Zope 2.5/2.6, hence we still stick with 2.3 for production sites until
all
bugs are fixed.

Cheers

Marc

> Date: Mon, 28 Apr 2003 10:41:36 -0700
> From: Scott Burton <scott@posplugin.com>
> Subject: Re: [Zope] Network Appliances NetCache vs. Squid
> To: zope@zope.org
> 
> Why do you really need to accelerate the site?
> If you have a highly dynamic site, no acceleration appliances will
work.
> There may be a few things you can do on your end first to see if there
is
> something easy to do to your instance of Zope.
> 
> Have you run profiling to see what is the bottleneck? Are there some
> expensive scripts that may need some reworking for better performance?
> Profiling can help with that. Do you efficiently use Zope's caching
> mechanisms like caching templates and images in ram, or increasing the
size
> of the ZODB cache? What OS is running Zope? Could you proxy Apache in
front
> of Zope on the same machine using mod_cache to speed up images etc.?
Could
> you build a simple 1u Linux box with ZEO and Squid on it as the
accelerator
> for less than a netappliance?
> 
> I would look into those things(if you haven't already) before worrying
about
> a caching/accelerator.

_______________________________________________
Zope maillist  -  Zope@zope.org
http://mail.zope.org/mailman/listinfo/zope
**   No cross posts or HTML encoding!  **
(Related lists - 
 http://mail.zope.org/mailman/listinfo/zope-announce
 http://mail.zope.org/mailman/listinfo/zope-dev )