[Zope-CMF] cmf zpt_generic/search - sequence_length is very slow for large collections, len () is much faster.

[Florent Guillaume]

Henryk Paluch  <hpaluch at gitus.cz> wrote:
>    I'm currently doing few scalability tests on Zope 2.7.3/Fedore Core 3 
> Linux 700MHz CPU, 640MB RAM, 2x IDE disks, CMF 1.5.0. The portal was 
> filled with 66000 portal Document objects (3 CMF BTreefolders, each 
> contains about 20000 documents). Fulltext indexes are realized using 
> TextIndexNG and storage is currently BDB. I started simple profiling of  
> document search (using log and DateTime() ;-) and find that for large 
> sets the expression
> 
> length = batch_obj.sequence_length
> 
> took nearly same taime as search itself, e.g.:
>  items = ctool.searchResults(kw)
> 
> For example - when search returns about 7000 items then 
> ctool.searchResults(kw) take about 30s, but
> the batch_obj.sequence_length took another 30seconds!
> 
> It tried to replace
> length = batch_obj.sequence_length
> with:
> length = len(items)
> ane wow! Now it is instant (about 100ms) - significant speedup.

They're not the same.

batch_obj.sequence_length returns the length of all the results.

len(batch_obj) returns only the length of the current batch slice.
Code is in ZTUtils/Batch.py

Because the search is lazy, to know the total number of objects you have
to find all the results, there's no escaping this.

> Also when tried search of Document objects only (no fulltext, eg.)
> http://z154.gitus.dom:8080/portal/search?review_state=&SearchableText=&Title=&Subject%3Alist=&Description=&created.query%3Arecord%3Adate=1970%2F01%2F01+00%3A00%3A01+GMT&created.range%3Arecord=min&portal_type%3Alist=Document&listCreators=
> Original code took 90s to display results, but modified version took just 3s
> 
> Can anybody confirm/explain this, or even submit this optimization into 
> CMF tree (if correct ;-)?

Your optimization is such that it changes the number of previous/next
batches displayed.

Florent

-- 
Florent Guillaume, Nuxeo (Paris, France)   CTO, Director of R&D
+33 1 40 33 71 59   http://nuxeo.com   fg at nuxeo.com