[Zope] minimizing conflict errors

[Chris McDonough]

On Nov 20, 2005, at 12:16 PM, Dennis Allison wrote:

> The structure of the naviagation method is simple enough.  
> Everything is
> wrapped in a <dtml-let> which sets a number of parameters mostly by
> reading them from the SESSION (with an interface function) or plucking
> them from the relational database with a query.
>
> In the scope of the let is dtml code which, when rendered, provides  
> the
> various navigation links.  In various sections there are additional
> <dtml-let> blocks and additional queries to the relational database
> and several <dtml-in> loops.
>
> Looking at the code, I don't understand why I am seeing conflicts.
> As I understand things, neither variables in the <dtml-let> space nor
> the REQUEST/RESPONSE space are stored in the ZODB so modifications to
> them don't look like writes to the conflict mechanism.  Am I incorrect
> in my understanding?

Yes, but that's understandable.  It's not exactly obvious.

The sessioning machinery is one of the few places in Zope where it's  
necessary for the code to do what's known as a "write on read" in the  
ZODB database.

Even if you're just "reading" from a session, looking up a session,  
or doing anything otherwise related to sessioning, it's possible for  
your code to generate a ZODB write.
This is why you get conflicts even if you're "just reading"; whenever  
you access the sessioning machinery, you are potentially (but not  
always) causing a ZODB write.  All writes can potentially cause a  
conflict error.

While this might sound fantastic, it's pretty much impossible to  
avoid when using ZODB as a sessioning backend.  The sessioning  
machinery has been tuned to generate as few conflicts as possible,  
and you can help it by doing your own timeout, resolution, and  
housekeeping tuning as has been suggested.  MVCC gets rid of read  
conflicts.  But it's not possible to completely avoid write conflicts  
under the current design.

Here's why.  The sessioning machinery is composed of three major data  
structures:

- an index of "timeslice" to "bucket". A timeslice is an integer  
representing
   some range of time (the range of time is variable, depending on the
   "resolution", but out of the box, it represents 20 seconds).    
This mapping
   is an IOBTree.

- A "bucket" is a mapping from a browser id to "session data  
object" (aka
   transient object).  This mapping is an OOBTree.

- three "increasers" which mark the "last" timeslice in which  
something was done
   (called the garbage collector, called the finalizer, etc).

The point of sessioning is to provide a writable namespace assigned  
to a single user that expires after some period of inactivity by that  
user.  To this end, we need to keep track of when the last time the  
user "accessed" the session was.  This is the point of the index.

When a user accesses his session, we may need to move his session  
data object (identified by his browser id) from one bucket  
(representing an older timeslice) to another (representing a newer  
timeslice).  This needs to happen *even if your code doesn't write  
anything to his session*, because it represents a session access, and  
the session is defined by total inactivity (not just write  
inactivity).  Likewise, when a user runs code that requires access to  
a session, but that user does not yet have a session data object, a  
write may need to occur.  So seemingly innocuous accesses to session  
data can cause a write.  Consider, in a Python script:

req = context.REQUEST
REQUEST.SESSION

Looks pretty harmless and unlikely to cause a write.  However, that's  
not true.  If the "bucket" in which the user's session data object is  
found is not associated with the "current" timeslice, we need to move  
his data object to the bucket that *is* associated with the current  
timeslice, which is a write operation in order to make note of the  
fact that his session is now "current".

Likewise with:

req = context.REQUEST
a = REQUEST.SESSION.get('foo')

Even though this appears to be "only a read", the sessioning  
machinery itself may need to perform a write operation to move the  
user's data object to the current bucket.

Jacking up the resolution time increases the period of time  
represented by a single timeslice, so fewer total writes need to be  
performed to keep a session "current".   Turning on "external  
housekeeping" doesn't prevent this normal movement of data objects  
between buckets, it just causes another process that cleans up  
"stale" data from happening during normal sessioning operations.

The sessioning machinery attempts to minimize conflicts.  The 2.8  
version of the temporarystorage does MVCC, which essentially  
eliminates read conflict errors.  The transience machinery includes  
significantly complicated logic to attempt to prevent conflict errors  
from occurring including code that attempts to prevent two threads  
from doing housekeeping at once as well as application level conflict  
resolution for simultaneous writes to the same session data object.   
However, the machinery uses BTrees to hold indexes.  BTrees also have  
a limited number of conflict avoidance strategies, but under certain  
circumstances (a "bucket split" is the canonical case) it cannot be  
avoided so not all write conflicts can be prevented without using a  
different kind of data structure to hold sessioning data.

A more detailed description of how "transience" works is available  
within the file named "HowTransienceWorks.txt" in the Products/ 
Transience package within Zope in case you're interested.

I hope this explains why you see conflict errors even if your code  
"doesn't do any writes", because actually it probably does by virtue  
of accessing a session.  Tuning the knobs that come with the  
machinery helps.  Causing transactions to be as short as possible  
also helps (by not using ZEO to back the sessioning database or by  
making your code just generally faster) because then there is less of  
a chance of a conflicting change.

- C