[Zope] The Principles of Designing a Heavy Load Site with Zop e

[sean.upton@uniontrib.com]

I'm chime in with some of my thoughts and questions...

Storage: why not something like logical volume management and a LV group
over a handful of DAS devices; seems cheaper than one big array?  I would
think 5 TB is doable on a 2-box cluster with 4-6 midrange DAS units
connected on dual-bus SCSI or Fibre per box, at likely less than $250,000 if
you shop for some bargains...  Then again, size may not be as much of an
issue as load?

Load balancing: L4 switch (or LVS?) in front of a group of Squid boxes in
front of a decent sized ZEO cluster.  Most L4 switches will do DSR/OPR
(direct-send or out-of-path return) instead of proxy mode, sending packets
directly from the balanced servers to the requesting client, but this packet
rewriting technique means that load-balancer cannot detect whether or not
the node its sending packets to is up, since it can't get a response, so you
usually have to use IP-takeover based clustering (Linux FailSafe or
Heartbeat) to takeover a dead proxy/caching server.  In theory, this
approach would eliminate a single load-balancer as a bottleneck because it
becomes a packet pusher and not a brokering proxy, which would make
something like this ideal for directing traffic to multiple proxy servers
under a high load... Price tag: 20,000 for 2x L4 switches, 20,000 for 4
decent Proxy boxes to run Squid.

Caching/Proxying: in response to the original post, I would think it would
make more sense to bypass Apache altogether and proxy to ZServer (with ICP
patches) directly from a competent load-balancing, ICP-capable, caching
proxy like Squid.  If nothing else, have Squid cache the page elements (all
static pages, images, client-loaded javascript source files) that can be
cached to alleviate some stress on your Zope server and allow you to still
keep things relatively simple.  Then again, I've never tried Apache+PCGI
with Zope...

Rendering: my question is when does a ZEO connection, for ODB-stored content
become the bottleneck?  I realize using a replicated RDB solves this problem
to some extent for those using ONLY relational data for content in their
apps, but I'm personally more curious about intensive use of ZODB. There are
issues of bandwidth to and processing power of a ZSS. Can ZEO (yet) support
using ClientStorage on an intermediary ZSS to use client caching in a
hierarchy of ZSS boxes?  I know that this wasn't technically possible a
while back, at least from what I remember reading on the ZODB-dev list, and
I wonder if it would actually yield better performance in most real world
scenarios?

Page Impressions & Rendering: I'm looking for an example of how Zope would
handle under extreme dynamic loads with a fairly straightforward
example...For a single Zope server (direct to ZServer), what kind of numbers
(Req/sec) would it be reasonable to assume that you could serve for a
otherwise static page that used a dtml-method that rendered results from a
Catalog query returning a dozen results (assuming direct reads from metadata
tables - no getObject() calls) on new-fast hardware (say, an Athlon 1.6GHz +
lots of DDR using FileStorage on a striped volume on fast disks)?  Could a
single box like this sustain 25-30Rec/sec in ideal conditions, assuming a
very aggressive use of aggregated HTTP keepalive connections was put in
front of it via proxy?  Personally, what I would like to be able to do is
eventually be able to serve 100 simple and uncached Req/Sec along these
lines from a 4-5 node cluster of ZEO Clients on 2CPU Athlon MP boxes running
2 Zope+ClientStorage instances each, with load distributed via Squid+ICP
(assuming that these requests were always a cache MISS in Squid)...

Sean

-----Original Message-----
From: Matthew T. Kromer [mailto:matt@zope.com]
Sent: Monday, March 04, 2002 7:49 AM
To: zope@zope.org
Cc: iap@y2fun.com
Subject: Re: [Zope] The Principles of Designing a Heavy Load Site with Zope


I think there are some fundamental misunderstandings going on here, but
I thought it would be interesting to try to respond anyway.


iap@y2fun.com wrote:

> Hi,
>
> This issue has been discussed again and again,
>
> I would like to clarify my idea and your comments will be very
> appreciated.
>
> Suppose we want to provide a server which is:
>
> 1) Hosting 1,000,000 members' profile. Each member's disk quota is 5MB.
>
> Which means we need at least 5,000GB (5 TeraGB) disk space.
>
> 2) Assume the concurrent accessing to a URL is 1000 request/per second
>
> 3) Assume all the requests retrieve dynamic content.
>
> 4) We want to leverage the power of Zope which means all the pages
> should be
>
> rendered by zope.
>


Having 5 TB of disk space usually means some very high-powered RAID
gear; my personal favorite is the EMC Symmetrix units; I think you would
probably want at least two of those to provide your coverage. Estimated
cost for this is about $5,000,000 (but *very* dependant on EMC's pricing
strategies).

You could get by for less, by distributing each disk with each CPU (the
breadrack approach.)

1000 requests/second isnt terribly high; Zope installations have done
400/sec with no problem. However, these are in situations where Zope is
being heavily cached; less than 10% of the requests are actually being
rendered by Zope. So, if you wanted no caching (ala everything is
completely dynamic content), my estimate is you would need something
like 100 1Ghz Intel Pentium III class machines to perform that amount of
dynamic rendering. If each of those machines had a 50 GB disk drive,
you'd theoretically have your 5TB of disk space.
At a rough commercial cost of $4,000 per unit (probably a bit high),
that's only $400,000.

As a practical matter, you'd then need some pretty hefty load balancing
servers; at least two, possibly more.

However, that begs the question of how you organize that much disk
space. It's not an easy task. Whether or not you use an RDBMS is
irrelevant until you can work out a strategy for using the disk space
scattered amongst all of those machines.

* * *

Now, if you forget for a moment about requiring each page to be
dynamically rendered each time it is viewed, and you set aside the
storage questions, you could estimate that, with a 90% caching rate, you
could serve 1000 requests/sec with only about 14 machines (10 renderers,
2 cache servers, and two load balancers). Estimated cost for that is
$56,000.

What is most unrealistic about this scenario is your assumptions about
the member base, and its ratio to expected activity. One million users
may only generate 1,000 requests/sec, but they certainly could generate
a lot more. In fact, a critical strategy for large systems like this is
anticipating for "peak demand" events. Lets say you send an e-mail out
to all million people, telling them to log in and check out a particular
URL. That timed event will generate a demand curve that is not evenly
distributed over time; in fact, it is usually very front-loaded. Within
about 5 minutes, more than 10% of the user base will respond.
This is a raw rate of about 333 requests/sec, but that prprobably esumes
that the
single URL is the only thing they load; usually, a page contains images
and other content (style sheets etc) which also much be fetched. Pages
with a high art content can have 25 elements or more on them. That
pushes the request rate up to 8333 requests/sec; way out of the 1000
request/sec bound.

> The priciples I would like to verify are:
>
> 1) Some database (RDBMS) should be used instead of FileStorage for ZODB.
>
> 2) The ZEO should be used for constructing a cluster computing.
>
> 3) The Apache should be the front end instead of ZServer.
>
> 4) The PCGI should be the connection between Apache and Zope.
>
> 5) I shouldn't create static instance into the ZODB but query the
>
> external database.
>
> 6) The "Cache" of zope is useless since all the responses are dynamic
> rendered.
>
> By the way, how much will this kind of system costs, regardless the
> hardware?
>
> Iap, Singuan
>


-- 
Matt Kromer
Zope Corporation  http://www.zope.com/ 




_______________________________________________
Zope maillist  -  Zope@zope.org
http://lists.zope.org/mailman/listinfo/zope
**   No cross posts or HTML encoding!  **
(Related lists - 
 http://lists.zope.org/mailman/listinfo/zope-announce
 http://lists.zope.org/mailman/listinfo/zope-dev )