[Zope] Re: [ZCommerce] Worthy group agenda: ZOPE dynamic pages and search engines... engines...

[Martijn Pieters]

>
>Hi,
>
>I guess this is not just a ZCommerce Problem, it's a general Problem of 
>dynamically
>websites, built for a shop or just for information use. Any project with a 
>lot of
>database stored content seems to have this problem. Maybe we can reach 
>some effect with
>providing META Information on dynamic pages and with the intensive use of 
>links. But
>for a general solution we need to know how robots work when meeting Zope 
>webpages, what
>they do and what not.
>I'm not that great with such Zope details, but maybe somebody else know, 
>how we can
>handle search engines for Zope.
>
>Joerg Herbst

Here is my limited knowledge:

I have found that search engines avoid URLs with query strings, so a URL like

   http://www.mynewssite.com/articles/retrieve?artid=163544

will never be indexed. However, 'normal' URLs, like:

   http://www.mynewssite.com/articles/artid/163544/article

which is, as some may note, a ZSQL query and display, _are_ indexed.

I don't have direct proof of the above, at least not with ZSQL Methods 
involved. However, any Digital Creations or Zope search on the major search 
engines will return URLs to www.digicool.com, www.zope.org, and 
zdp.zope.org, all websites running on Zope.

I think, that as long as we can keep query strings out of the URLs that we 
want to be indexed by spiders, things will be just fine.

A small test:

ZDP hosts the ZBook, and this can be read chapter by chapter, or as one 
long page. The chapter by chapter version uses a tree navigation widget, 
and as such uses query strings. The long page doesn't use a query string. 
Both versions are dynamically created, as is the rest of the ZDP site.

Searches on Altavista and AllTheWeb on 'zdp zbook zen' immediatly return a 
link to the long page (http://zdp.zope.org/guide/ZBook/zb_all) but failed 
to return any of the tree-tag pages.

Similarly, searches for zope faq return very relevant links.

--
Martijn Pieters, Web Developer
| Antraciet http://www.antraciet.nl
| Tel: +31-35-7502100 Fax: +31-35-7502111
| mailto:mj@antraciet.nl http://www.antraciet.nl/~mj
| PGP: http://wwwkeys.nl.pgp.net:11371/pks/lookup?op=get&search=0xA8A32149
------------------------------------------