[Zope-dev] [Interested?] New use for Zope+CMF+Archetypes -- startup time improvements

[robert rottermann]

Dieter,
this sounds very interesting.
The idea of creating  such application that are "shared" between desktop 
and web has always appealed me and I was wondering how to implement it.

So I would be very interested
- that your patches make their way to the Zope core (I am not a Zope 
core contributor tough)
- to learn more about what you have done.

Robert

Dieter Maurer wrote:
>Dear Zope developers,
>
>we have used Zope+CMF+Archetypes in a new way -- not
>as a Web Application framework but as a framework for
>desktop applications that share a large part of their
>functionality with online applications (implemented with Zope+CMF+Archetypes).
>
>A major stumbling block has been Zope's incredibly high startup
>time. We observerd times in the order of a minute on computers
>with either slow CPU or slow IO. This may be acceptable for
>a Web server but is prohibitive for a desktop application -- especially
>as the predecessor application started within a few seconds.
>
>To overcome this obstacle, we tweaked Zope and fixed Python's import
>mechanism such that Zope now starts either out of a ZIP archive
>or as a frozen application. These measures had the following results.
>
>  Startup times on a mid range computer (AMD Athon 1.4 GHz; 512 MB memory)
>  with a standard IDE disk.
>
>		    Cold start			    Warm start
>		(after computer startup)	(most files in OS cache)
>
>    File system	       13s			      5s
>    ZIP archive	        8s			      4s
>    Frozen		5s			      3s
>
>
>In more details, we did:
>
>  *  implement a package for a new kind of "url"s "pypackage:"
>     for package relative access to resources.
>
>     The package monkey patches Python's "open", "os.listdir",
>     "os.stat" to provide transparent access to
>     "pypackage:" identified resources.
>
>     It currently support package relative access for
>     packages loaded from the file system, from a ZIP
>     archive and from the executable itself (i.e. frozen packages).
>     In the last case, the resources are in a separate ZIP
>     archive.
>
>     This package might be interesting for Python as a whole
>     as it is not Zope specific.
>
>  *  implement a shared object importer to be used
>     as a Python "meta_path" hook.
>
>     This importer allows to load shared objects into the context
>     of a parent package (such as e.g. "ZODB.TimeStamp") although
>     the shared object is not located inside the package's source
>     (ZIP archive or executable).
>
>  *  fix about 70 occurrences in Zope code where
>     package relative access was implemented by "dirname(__file__)"
>     to consistenty use "package_home".
>
>  *  modify about a dozen places in Zope+CMF to use
>     "pypackage:" and cope with "__path__" not being a list
>     for frozen packages
>
>  *  fix a few products (Archetypes and friends, TextIndexNG2,
>     PlacelessTranslationService, ...) to use
>     "package_home" (rather than "dirname(__file__)") and
>     not to change the current working directory (which obviously
>     would fail for destinationsbe in a ZIP archive
>     or the executable).
>
>  *  implement lazy loading of "ImageFile"s to
>     reduce the risk of recursive imports (and reduce startup time).
>
>  *  support lazy product initialization
>
>  *  support configuration from a pickle file (to avoid
>     expensive parsing of the schema and configuration files).
>
>     The pickle is used as a configuration cache.
>
>  *  fixed Python's import mechanism not to treat ZIP
>     archives as a directory when the archive could not
>     find a module.
>
>
>If you were *really* interested in these startup time improvements,
>I could provide patches which might be integrated
>in the Zope core for e.g. Zope 2.9.
>
>
>