[Zope] Re: Index Problem - Index randomly breaks/changes

Ryan Smith rsmith985 at gmail.com
Thu Mar 6 11:46:21 EST 2008

On 3/6/08, Tres Seaver <tseaver at palladion.com> wrote:
> Ryan Smith wrote:
> > Posted this to the Plone Mailing-list, but this is more of a problem I
> > am having with Zope. There will be additional information here -
> >
> http://www.nabble.com/Indexing-Bug---Index-changes-and-is-not-always-correct.-td15830136s6742.html
> >
> > Basically the problem I am having is that I create an index, but
> > sometimes this index changes and is incorrect.
> >
> > To test this I would constantly click the 'Clear and Rebuild' button
> > under the Advanced Tab.
> > Then I would browse my index under the Indexes tab.
> >
> > After repeating this many many times, I would get 2 different results
> > for my index (1 correct, the other completely wrong). I didn't change
> > anything else, I just kept clicking rebuild.
> We need more information:
> - What kind of index?
> - What objects are you indexing?
> - What kind of query are you making against the index?
> - What does "completely wrong" mean?

Here are the 2 word answers, I go into more detail below.
 - What kind of index?
FieldIndex on an attribute I created called 'access'.  Access is a
list of accesses a user has.

 - What objects are you indexing?
Any object with the 'access' attribute

 - What kind of query are you making against the index?
I noticed the problem when doing a folder listing in my Plone Site.

 - What does "completely wrong" mean?
The index is missing objects that have the 'access' attribute.  If an
object's 'access' attribute is B it may be mapped in the index to A.

Here is some background on what I am doing -
For a plone site, I needed a way to allow a user to only see objects
if they had a certain set of roles.  I can't use the default security
policy because we may want an object to require the roles Alpha AND
Beta, but by default if a user has Alpha OR Beta they can see the

When creating an object, it is possible to specify and 'access'
attribute that will require special accesses to view the object.

I handle if a user can view an object by dynamically adding them to a
group, via a Group Plugin.

Problem was this didn't effect if the object showed up in a search
result, like a folder listing.  Plone calls ZCatalog.searchResults()
to get the metadata for all the objects in a folder.  I grab these
results and remove objects that shouldn't be listed to a specific
user.   To check if a user is allowed to view it, I check in the
metadata, if my attribute is there I compare it to the users roles and
remove it if needed

This all works most of the time.  But sometimes the index changes and
the results I filter out will be wrong.

> > I noticed something interesting in my debugging window.
> > All rebuilds that resulted in a correct index had this -
> > DEBUG txn.-1223402608 commit <Connection at 0b5a01ec>
> > All rebuilds that resulted in an incorrect index had this -
> > DEBUG txn.-1244382320 commit <Connection at 0965020c>
> >
> > Something different is being done, I'm just not sure what or why.
> >
> > Any suggestions as to why this is happening or where I should look to
> > fix this would be greatly appreciated.
> The two log messages you post are effectively identical: some get
> served from different ZODB connections, but that shouldn't cause any
> difference in behavior. If you have one connection with "stale" data
> for some objects (that would be a bug somewhere), then restarting Zope
> might make the problem disappear, although we still wouldn't know how
> that connection got into that state.

The point I was trying to make with this was that if I reindex 100
times.  I will get "DEBUG txn.-1223402608 commit <Connection at
0b5a01ec>" 80 times and "DEBUG txn.-1244382320 commit <Connection at
0965020c>" 20 times.  I realize they are nearly the same, but I get 2
distinct values every time.  So if I dont restart Zope, it will always
be one of these 2, nothing else.  And the re-indexes that are wrong
always share the same value.

More information about the Zope mailing list