[Zope] Poor man's (application level) database replication?

[Steve Spicklemire]

Hi Kevin,

Thanks for the feedback! Your system sounds a lot more sophisticated 
than what I was planning. My assumptions are basically:

1) Failure is very rare, but over long periods of time, it is likely 
there there will be some failure,

2) There are lots of folks out there who want to have a more robust 
setup, but can't afford to buy expensive solutions, or become experts in 
the more arcane issues of extreme reliability.

The question is, when a failure does occur, is there any (low cost, 
easy, simple) way to mitigate its effect?

I think that the idea of a 'policy' object could in principle handle 
"3", but I'm thinking that it probably mostly won't do any of these 
things. At least not right away. I'm thinking that

1) Existing DB adaptors already handle caching (I haven't looked into 
how they handle invalidating their internal cache.. but I suppose I 
should!).

2) The primary issue I'm worried about is not performance (though I 
don't want to incur a huge performance loss!) but increasing the 
likelihood that a database failure will not bring down the system. I 
think that load balancing would be nice, but something much cruder 
(i.e., each ZEO client uses a different DB server for reading) will also 
help performance, if not optimize it. (Note that if you load balance the 
ZEO clients, using TD's ICP code, or some other method, at least for 
database bound applications, it may well have a follow-on database 
load-balancing effect!)

Anyway.. I'm encouraged enough to take a first stab at this.

thanks!
-steve


On Friday, May 31, 2002, at 07:39 AM, Kevin Carlson wrote:

> Steve,
>
> Interesting idea.  Several years ago (OK, a lot), prior to the 
> commercial
> availability of reliable replication of non-mainframe databases I wrote
> something similar to this in C.  The approach you recommend is very
> similar -- here are a few of the things that my version of the db-tee 
> did:
>
> 1) Kept track of the network time or metric to reach each db.
> 2) Had a running reference count of active threads using the instance 
> of the
> tee
> 3) Used a load-balancing technique taking the prior two items into
> consideration
> 4) Cached write operations (see below).
>
> One of the problems we ran into early on was the issue of 
> cross-corrupted(?)
> caching.  What I mean is that any caching you're doing at the unique db
> level is effectively unusable unless you build in the ability to blast 
> out a
> "record invalidated" message to each cache to allow for the removal of 
> any
> recently changed or deleted rows.
>
> An additional challenge that is faced is that of single machine failure.
> When a single machine fails, it does not receive the ongoing write
> operations from the tee.  Therefore, when it is brought back up again,
> things are usually way out of sync. One option is allow an agent (in 
> this
> case, perhaps the policy object) to cache all write operations and 
> ensure
> they are executed at each adaptor instance.
>
> Anyway, I think the idea is timely and challenging.  My hat's in the 
> ring if
> you need help designing and implementing this.
>
> Kevin
>
> -----Original Message-----
> From: zope-admin@zope.org [mailto:zope-admin@zope.org]On Behalf Of Steve
> Spicklemire
> Sent: Thursday, May 30, 2002 5:17 PM
> To: zope@zope.org
> Cc: Steve Spicklemire
> Subject: [Zope] Poor man's (application level) database replication?
>
>
>
> Hi Folks,
>
> This might be the world's dumbest idea... but I'm close enough to
> thinking that it's "OK" to start wasting time on it, and I want to get
> some feedback before I actually waste very much time. ;-)  It goes like
> this: I need to be sure that my SQL data is replicated somewhere.
> Replication seems to have spotty support among the free database, and
> it's also fairly complicated to set up. Mere mortals would like
> something easy, and dumb, to help out in cases where these two obstacles
> are a significant hurdle. ZSQLMethods required you to choose a database
> adaptor, and database adaptors generally permit you to connect only to a
> single database. I'm thinking that I could cook up a simple "db-tee"
> adaptor that keeps track of two things:
>
> 1) a list of "real" database adaptors that it knows about and their
> current "status" (OK/broken)
> 2) a callable "policy" object (python script, external method, ZClass?)
> that can work with it's list of DAs to implement the replication by
> directing the iteration process.
>
> The basic plan is that a ZSQLMethod attaches itself to the tee, and
> calls the tee for database access. The tee then uses it's policy object
> to iterate through the list of database adaptors. For read operations
> (e.g., SELECT) the policy would use a "continue until successful"
> strategy, while write operations would use a "iterate over all DAs"
> strategy. Even the choice of which adaptors to iterate over could be
> variable (e.g., depend on the value of ZEO_CLIENT or somesuch). This way
> reading from the database(s) would be faster (since each ZEO_CLIENT
> could use a different DA as the first DA to check) but writes would be
> slower (since each DA would need to be called in turn to get
> replication). But, in my experience, writing is much less frequent than
> writing, so maybe that's OK? Also, the policy object could decide how to
> handle failures (on read: skip and set status to "broken", on write:
> raise an exception, or notify someone, or whatever you like).
>
> Anyway.. I'm just wondering if anyone has toyed with this sort of plan,
> or has come up with something better?
>
> thanks!
> -steve
>
>
>
> _______________________________________________
> Zope maillist  -  Zope@zope.org
> http://lists.zope.org/mailman/listinfo/zope
> **   No cross posts or HTML encoding!  **
> (Related lists -
>  http://lists.zope.org/mailman/listinfo/zope-announce
>  http://lists.zope.org/mailman/listinfo/zope-dev )
>
>
>
> _______________________________________________
> Zope maillist  -  Zope@zope.org
> http://lists.zope.org/mailman/listinfo/zope
> **   No cross posts or HTML encoding!  **
> (Related lists -
>  http://lists.zope.org/mailman/listinfo/zope-announce
>  http://lists.zope.org/mailman/listinfo/zope-dev )