[Zope3-Users] zc.tables integration with Zope3

[Jim Washington]

David Pratt wrote:
> Hi Jim. This approach is completely new to me. It is interesting and 
> I'll keep an eye on your blog. How expensive is updating indexes with 
> data modification. You are talking about sqlite, does this mean you 
> figure the indexes would be a significant drag on the ZODB due their 
> size or is this idea a performance consideration. Many thanks.
>
The expense is in recalculating the new factoradic for each stored 
index, which involves a lot of division and factorial operations.  But 
this might be done in a smart way.  Maybe instead of on-change, we could 
recalculate an index on first access to that index after change.

Sqlite would be mainly for performance, ATM.  A lookup is less expensive 
than calculating a factoradic of a very large number.

Size may also be an issue if the index is stored in the zodb.  An 
integer (long as it might be) is the most compact representation of the 
factoradic. I'm pretty sure that longs end up as base ten strings in a 
pickle, but that is still smaller than the pickle of the factoradic list 
representation.  I still need to experiment more, but it's not out of 
the question that the indexes could end up as sqlite-backed annotations 
to the container/data source, particularly when sqlite may already be in 
the mix for memoizing factorial and/or factoradic calculations.

It may still turn out that this is all not worth the effort in practice.

-Jim Washington