[Zope] Hardware for Zope + ZEO
Wankyu Choi
wankyu@neoqst.com
Fri, 31 Jan 2003 15:36:59 +0900
From: Mario Valente [mailto:mvalente@ruido-visual.pt]=20
Sent: Wednesday, January 29, 2003 10:38 PM
To: Wankyu Choi; zope@zope.org
Subject: Re: [Zope] Hardware for Zope + ZEO
First of all, thanks for your detailed and informative feedback. I could
hardly find any feedback regarding this issue for months.
>> We had a similar setup (IIS + ASP + MSSQL) that was crashing at =
least
once a day. Once converted to Linux+Zope+MySQL it ran happily on the =
same
hardware (we actually took out dual processor frontends and put in =
single
processor frontends and there was no problem) and had uptimes of 70 =
days.
My servers have been up and running for a year and a half :-) I'm =
pretty
much satisfied with the existing APM setup. The only reasonly I'm =
switching
to Zope/Python is ... for better application productivity. What I could =
do
in an APM setup, I could do it in 1/10 time and efforts. With these ZODB
storage worries gone, I have no complaint about Zope/Python ;-)
>I plan to buy one No.1 machine plus one No.2 machine since I already=20
>have two machines, candidates for the other two ZEO clients.
>
>> OK, although I think that since those machines are multiprocessors
they might be put to better use.
The problem is the existing machines should still be serving users while =
I
toy around with the new machines. That is, I plan to slap together a =
beta
testing environment with the new machines installing everything from =
scratch
while the existing ones keep running. I'll need at least two months to
finish the setup and necessary Zope/CMF/Plone applicatioins like =
NeoBoard
and NeoPortal Content Pak and more.
>I came under the impression using Zope for the past year ( and from=20
>python/zope docs ) that python uses only one CPU no matter how many I=20
>have. One of the two machines I already have has four Xeon cpus but=20
>Zope on that machine runs way slower than one on my single P3 desktop=20
>with a bit more horsepower.
>
>> I would use this machine as a backend server (with an upgrade to =
the
disks and RAM perhaps), saving the expense of buying machine nr 1.
Like I mentioned above, it should keep running while I set up the other
machines. When the two new machines are properly setup and my =
application
development projects are complete, I'll go public with these machines =
and
take down the existing ones; clear up the mess in the old machines ( =
running
too long with old OS stuff ) reinstalling everything from scratch as I =
will
have done with the new machines; put them back in service. The number 4
machine with 4 CPUs will also be put to use as a mail server as it is =
now
for example. So I won't be wasting their horsepower. I run serveral
mailinglists with an average 100,000 subscribers, for one thing ;-)
This might look like a stupid plan... but whenever I did a major =
overhaul,
something went wrong. About this time of the year...2002... I even =
tried an
automatic setup process of my own creation with a tons of shell scripts =
and
tested them on my Linux box again and again before actually running them =
on
the production machines: an hour of a job, I thought. But it took a day =
and
a half ( no service was available; users were screaming down my neck ) =
Never
did I dream that RedHat wouldn't install on an intel 440 machine with a
particular adaptec scsi controller; it just hung. Hours of searching the
Redhat Bugzilla turned up an entry only days old regarding this issue. I
even passed out after that day and a half of nightmare :(
With two months of leeway and lastest models of servers, I won't have to
worry about such nonsensical, unexpected glitches ;-)
>Would this setup seem reasonable? Or better still, how would you set=20
>your portal up if you got extra dough for hardware?
>
>> If you intend to spend some extra money, spend it on a front end =
load
balancer caching system. You would put this in front of your ZEO client
machines, spreading the load and caching most of the pages.
Can't I just use linux virtual server? Seems like a lot of waste to buy =
an
expensive load balancer for so few a machines...=20
Zope will be running behind Apache + Squid fronts.
Bad ideas? Just a thought, haven't tried it.
>2. Some say P3 tualatin is better for Python than P4 or even Xeon=20
>processors. Is that true?
>
>> I'm not aware of this. But even if true, your machines wont be =
running
Python only. They'll be running the OS, system tasks, etc.
I meant a ZEO client machine. I don't plan to hog its CPU with any other
serious tasks. The other two exsting machines already have multiple CPUs =
and
they'll double as a mail server and stuff.
>> I definitely woulnd use FileStorage. Mainly because when Data.fs =
grows
big, Zope startup times go to hell. We're using BerkeleyDB Storage with
great success. You can either run it versionless or not. And its quite =
fast.
>> We've been looking at DirectoryStorage but right now its still not =
final
and we're afraid that at some point (ie large sites with lots of =
objects)
the OS might run out of file handles.
Thanks for bringing the BDB Storage to my attention. ( ReiserFS has no =
file
handle/inode problems, though. )
>> Also keep in mind that dual processor systems help in eliminating
single points of failure. You woulnt want to have a nice setup with a
single processor backend server and have that processor fail :-)
Hm... that's a good point. I haven't thought of that :-) Taking out a =
CPU
from the total cost of buying a server doesn't even make a dent anyway =
:(.
>Machine No.4 - 4 Xeon CPUs (older model) + 2G mem + 4 36G disks (disks=20
>old, will be recycled for backup storage or something)
>
>> Use this as your backend server (MySQL + ZEO). Use BerkeleyDB or
DirectoryStorage. You should only store in Zope what is "Zope stuff" =
(DTML
methods, Python methods, ZClasses, ZSQL queries, etc). All the original
site data should be kept in MySQL. All images should be stored as =
External
Files (as well as any other BLOBs like PDF files, DOC files, etc).
which means... you wouldn't put any serious heavy-weight data into your =
ZODB
storage no matter how good it might be?=20
That's the problem I'm not exactly sure about. For example, I have tons =
of
textual data stored in MySQL via PHP NeoBoard ( a web-based discussion =
board
). I rewrote NeoBoard from scratch as a Zope product. It stores all its =
data
in ZODB. I was thinking of pulling all the existing articles into ZODB =
to
use Zope NeoBoard. Now the plan looks like a silly idea... I seem to =
have to
do just the opposite: take out all data stored in ZODB and put it into
MySQL. That way, Zope NeoBoard Article objects will only act as wrappers
pulling acutal data from MySQL tables.=20
Plus, I rewrote all the CMF content types to add extra features and =
stuff.
Now I'll need to do the same with these content types to make use of =
MySQL.
Is my assumption right? That makes ZODB look like a metadata holder, not =
a
database :(
>> Hope this helps.
It surely helped. Thanks a lot.
Cheers,
Wankyu Choi
---------------------------------------------------------------
To the dedicated staff at NeoQuest, language is not a problem
to be dealt with, but an art waiting to be performed.
---------------------------------------------------------------
Wankyou Choi
CEO/President
NeoQuest Communications, Inc.
3rd Floor, HMC Bldg., 730-14, Yoksam-dong, Kangnam-gu
Seoul, Korea
Tel: 82-2 - 501 - 7124 Fax: 82-2-501-7058
Corporate Home: http://www.neoqst.com
Personal Home: http://www.zoper.net, http://www.neoboard.net
e-mail: wankyu@neoqst.com
--------------------------------------------------------------- =20