[Zope] Faster ZCatalog Searches (better? Optimized?)

[Casey Duncan]

Doing *any* sorting at the DTML level from the Catalog will kill you simp=
ly=20
because ZCatalog returns a lazy result set. Sorting it in DTML involves=20
touching every element of the lazy result. This is very expensive both in=
=20
time and memory.

You should let the Catalog sort this for you using its indexes. Like so:

<dtml-let results=3DsearchResults(REQUEST, in_reply_to=3D('',),  sort_on=3D=
'id',=20
sort_order=3D'reverse'))>
=2E..
</dtml-let>

hth,

Casey

On Monday 29 April 2002 07:09 am, BZ allegedly wrote:
> Hey Toby,
>=20
> I played around with this way too much last friday and the=20
> performance hit, from what I can tell, is having the dtml do the=20
> reverse. I looked at code from Zope.org and they do it this way:
>=20
> <dtml-with "_(results =3D searchResults(in_reply_to=3D[''], REQUEST=3DR=
EQUEST))">
>=20
> If I take out any of the sorting, I can get 10x the speed, but at=20
> that point, what is the point?
>=20
> BZ
>=20
>=20
> At 10:45 AM +0100 4/29/02, Toby Dickenson wrote:
> >On Fri, 26 Apr 2002 14:26:39 -0400, BZ <bz@bwanazulia.com> wrote:
> >
> >>ZUBB:
> >><dtml-in "searchResults(in_reply_to=3D[''])" size=3Dbatch_size revers=
e
> >>sort=3Did start=3Dquery_start>
> >
> >You are are getting dtml-in to do the sorting, not the catalog. The
> >difference may be very small, or very large
> >
> >Try this (untested) code as a perfomance test
> >
> ><dtml-in "searchResults(in_reply_to=3D[''],sort_on=3D'id')"
> >size=3Dbatch_size
> >start=3Dquery_start>
> >
> >That should be faster, but reverses the sort order. In not sure if
> >putting the 'reverse' dtml-in tag back in will incur a performance
> >penalty.
> >
> >
> >I have recently committed some significant micro-optimisations to the
> >the catalog for sorted queries, which have improved the runtime for
> >some of my queries by a factor of 8. If you want these, they are
> >currently in the CVS trunk and will definitely go into 2.6.
> >
> >
> >
> >Toby Dickenson
> >tdickenson@geminidataloggers.com