[Gsoc] Respector: A Grok Introspector Replacement

[Lennart Regebro]

A quick look through looks like I agree with most of this. :)
Basically what is needed is a component registry viewer and an object
inspector that both are pluggable enough so that frameworks like Grok
and Plone can plug in their own stuff. Integration with some
apidoc/docfindertab thingy would be nice too.

The component registry helper we hopefully has somebody to do.

Just my 2 centimes.

On Thu, Mar 27, 2008 at 4:28 PM, Uli Fouquet <uli at gnufix.de> wrote:
> Hi there,
>
>  I followed the interesting discussion on the list. Especially
>  ZODB/registry/component browsers seem to be a hot thing currently ;-)
>
>  Because my last years' work on the Grok introspector is affected at
>  least indirectly by that development and because I am considering to
>  apply one more time for GSOC, I would like to put in some thoughts
>  concerning the Grok introspector, but also all kinds of 'runtime
>  scanners'.
>
>  As usual, please forgive the length!
>
>  Respector: A Grok Introspector Replacement
>  ******************************************
>
>  The Grok introspector I developed during GSOC 2007 is still missing
>  some fundamental functionality and suffers from some fundamental
>  design flaws.
>
>  After (re)thinking about it and reading all the ideas mentioned in
>  this list (most notably: ideas collected and put in by Lennart and
>  Martin Aspeli), I came to the conclusion that a proper
>  reimplementation might be the best way to get a more powerfull tool
>  here. A reimplementation, that is based on a more general engine.
>
>  This might inlude improvements on the ZODB broken object model.
>
>
>  What does 'respector' mean here?
>  =================================
>
>  The word 'respector' is built from 'replacement', 'debugger' and
>  'introspector'.
>
>  It denotes both of the (currently) two distinct parts of 'runtime'
>  orientation for users in Grok, the ZODB browser as well as the
>  package/component browser. While the first one gets and displays
>  information about ZODB objects, the latter delivers (or: should
>  deliver) information about loaded packages, interfaces, directives
>  etc. It should be kept in mind, that ZODB and registry objects might
>  be only a part of the things users might want to examine in a runtime
>  environment.
>
>
>  Problems with current approach
>  ==============================
>
>  The current approach of `docgrok`, the tool that does introspection
>  for Grok, suffers from some shortcomings:
>
>  - it is bound quite strictly to Grok, which means: it requires a full
>   fledged Grok instance running to work.
>
>  - it makes strong assumptions about the environment it runs in, namely
>   the existance of `apidoc` and other voluminous packages.
>
>  - it comes as a part of the grok core, which bloats the number of
>   packages needed to run grok, even if the introspecting functionality
>   is not wanted (for example in production environments).
>
>  - it might be considered a security risk, because it frankly passes all
>   information to (authenticated) users without a possibility to
>   restrict access. There is no possibility to create something like a
>   'developer user'.
>
>  - it does not provide an interface other than the Web-UI, nor does it
>   provide entry points for, say, remote debugging without a browser.
>
>  - its API might be too complex. While it is possible to register
>   arbitrary types of things to be displayed by `docgrok`, the way to
>   do it might be plainer.
>
>  - it makes too little use of other tools already available (basically
>   it wraps only `apidoc`).
>
>  Apparently `docgrok` also lacks the ability to display important data
>  like directives used in a view, available layers, default values
>  etc. This, however, is more a question of diligence, because in
>  principle all this is supported by the current approach. It would only
>  need a concrete implementation for concrete types of things.
>
>
>  Renewing the Grok admin app
>  ===========================
>
>  Some of the shortcomings mentioned above can be tackled by simply
>  factoring the admin-UI out of the Grok core/trunk. Such, a user can
>  decide whether s/he needs this functionality or not on his/her own.
>
>  To cope with the deeper design flaws, I can think of a two-step
>  strategy:
>
>  1) factor out introspection stuff from the then standalone admin app
>    into an own package or subpackage that can be used also without
>    grok/grok.admin.
>
>  2) give it a new, more general, easy to use base.
>
>  All this should not be too complex, except the 'general base' stuff.
>
>
>  A more basic foundation for grabbing runtime information
>  --------------------------------------------------------
>
>  Reading all the proposals and ideas on the grok-dev and GSOC lists I
>  see, that there is a high demand for a better integrated
>  'runtime-scanner' which can be used in different environments (Plone,
>  Grok, etc.) and with different frontends (webbrowser, shell, wxpython,
>  etc.).
>
>  If such a tool exists, also Grok should of course make use of it.
>
>  From the experiences from last GSOC I might mention some things to pay
>  attention to when creating such a tool.
>
>  A basic runtime scanner
>
>  - should be extensible in the way, that it can handle different kinds
>   of data sources such as the ZODB, registry etc. It should be
>   possible to implement new scanners based on a well-designed API if
>   appropriate, for example to add support for RDBMs one day.
>
>   This means, that in the beginning a basic scanner would be needed,
>   that makes nearly no assumptions about existing things in the
>   environment, but is extensible to provide support for example for
>   scanning the ZODB. The zodbscanner in this case would be a special
>   case of the base scanner.
>
>  - should separate the tasks of information retrieval and information
>   presentation. Nearly all tools existing in that area do that
>   internally, namely `docgrok`, `apidoc` and `zodbbrowser`. But they
>   all make assumptions about the used frontend (browser, wxpython-GUI)
>   and offer only support for that kind of 'display'.
>
>   A well designed runtime-scanner might be prepared for collaboration
>   with unknown presentation 'plugins'. The separation of retrieval and
>   presentation would ease the implementation of specialized frontends
>   for Plone, Grok, etc.
>
>  - might be prepared for modification of objects. This means, it might
>   provide a 'communication channel' to tell a frontend, whether a
>   certain attribute/object/thing is modifiable and perform this
>   modification on request.
>
>   This way, the scanner would turn into a debugging tool.
>
>  All this is _much_ work, probably too much for a single GSOC, but at
>  least a plain but well thought base implementation should be possible.
>
>  When the base API exists, numerous scanner extensions/frontends could
>  be implemented independant from each other later on.
>
>
>  A renewed runtime scanner for Grok
>  ==================================
>
>  Retrieving information
>  ----------------------
>
>  Given that such a general base exists, Grok certainly needs a
>  specialized version that diggs for
>
>  - directives used (also default ones, not explicitly used)
>
>  - views/layers (also ones not currently used but available)
>
>  - other stuff
>
>  This would basically mean to implement a scanner, that extends the
>  base scanner, because every scanner can only look for things it is
>  aware of. This might make use of existing tools like `apidoc` to
>  avoid reinventing the reel.
>
>
>  Presenting information
>  ----------------------
>
>  Once, the scanner works, an appropriate 'frontend' would be needed,
>  that naturally would be a set of views.
>
>  For Grok there might be much code of the current docgrok
>  implementation that can be reused for that.
>
>  I could also think of a wxpython-based remote tool for debugging,
>  which would require an appropriate 'frontend' as well.
>
>
>  Modify information
>  ------------------
>
>  It would be nice to be able to change values of certain objects
>  on-the-fly.
>
>  A nice feature (not only for Grok) would be the ability to edit broken
>  objects. This would require a generic broken objects implementation as
>  proposed on http://wiki.zope.org/gsoc/SummerOfCode2008. I already
>  implemented support for deletion of broken objects in Grok.
>
>
>  Is all that necessary?
>  ======================
>
>  No, it's not. For Grok we can archive the same functionality with the
>  existing docgrok stuff. It, however, makes sense from my point of
>  view, because a unified runtime scanner would distribute the effort for
>  keeping everything up-to-date on many shoulders and avoid
>  double/triple work. Why should the same ZODB scanning be performed by
>  four different packages, when in fact they all try to grab the same
>  information? Unfortunately, this is the current state.
>
>  Maybe something like a base runtime scanner already exists due to
>  efforts starting at the Plone summit. The apidoc package is usable
>  anyway and also the z3c.zodbbrowser might be of use, if it would
>  separate the information retrieval and presentation (currently wxpython
>  based, right?) even more accurate.
>
>  The Grok admin UI needs a redesign anyway to address all the issues
>  mentioned above. So why not go back to the roots?
>
>  Furthermore this addresses a topic, which crosscuts several planned
>  projects for upcoming GSOC. Why not try a collaborative effort? Is there
>  a better opportunity?
>
>  Kind regards,
>
>  --
>  Uli
>
>
> _______________________________________________
>  Gsoc mailing list
>  Gsoc at zope.org
>  http://mail.zope.org/mailman/listinfo/gsoc
>
>



-- 
Lennart Regebro: Zope and Plone consulting.
http://www.colliberty.com/
+33 661 58 14 64