[Zope-dev] the Zope Framework project

[Martijn Faassen]

The Zope Framework project
==========================

:Author: Martijn Faassen
:Date: 2009-03-02

Introduction
------------

This document offers suggestions to reorganize our community so we can
act more effectively. It does this by trying to clarify what our
community is about. The document tries to innovate minimally in
concepts and naming in order to provide a relatively small
evolutionary step forward that can still make us all work together
better. Even though this is an evolutionary step, it will still have a 
big impact if implemented, so please read on.

This document should be relevant to *all* the parts of our community
that build web applications, whether they use Zope 2, Zope 3, Grok,
Repoze, or applications built on top of these such as Plone or
Silva. While it talks a lot about Zope 3 this is because the Zope
technology within Zope 3 is used within all these projects. The
document wants to recognize this officially.

The main innovations in concepts are the name "Zope Framework" to
distinguish it from the Zope 3 application server and the
"core"/"extra" concept. These are all hopefully descriptions of what
are current practices, simply making them more explicit.

The biggest innovation is the introduction of a Zope Framework
Steering Group as a new entity that will be the steward for the
development of this framework. The steering group's primary aim is to
facilitate developers in the community so that they can continue to
maintain and develop the framework so that it is useful to all of us.

In preparing this document I've shown previous versions of it to
various members of our community. These people have generally endorsed
this effort and given me feedback, which encouraged me to go forward
with it. I've edited it further since then, so this is still my
proposal and is not automatically endorsed by anyone else. I hope of 
course that they will reply to endorse this version now!

I now make this document public to:

* gather more feedback.

* let everybody get used to these ideas.

* get a positive response so we can move forward with these proposals
   soon.

I hope we can start implementing these suggestions by the end of march
or sooner.

The Zope project
----------------

Before we can deconstruct the Zope 3 project, we will look at it
in the wider perspective of the Zope project as a whole.

What is the Zope project? The Zope project is an umbrella project for
a number of sub-projects. Its source code is in a repository managed
by the Zope Foundation. Within the Zope project, there are a number of
projects that ship full-stack web frameworks (or application servers):

* Zope 2 application server

* Zope 3 application server

* Grok

Besides full-stack web frameworks, the Zope project also maintains a
lot of libraries such as the ZODB, Zope 2 products, and a lot of Zope
3 libraries. The project also maintains the buildout system.

Some projects are based on Zope technology but are not maintained in
the repository of the Zope Foundation. These are not considered Zope
projects but can nonetheless be important projects to the Zope
community. Examples of such important consumers are Plone, Silva and
Repoze.

Deconstruction of Zope 3
------------------------

For a long time the community has had a project named "Zope 3". This
project actually does two things:

* maintain and evolve the reusable Zope 3 libraries.

* maintain and evolve the Zope 3 web application server.

It's clear Zope 3 currently has a dual nature. It is both considered
to be a collection of libraries that other projects use and build on,
as well as a particular full-stack web framework with a particular
user interface (ZMI).

There is a community consensus that the perspective on Zope 3 as a
collection of libraries is the most central one, as there are a lot of
consumers of some of these libraries beyond the Zope 3 application
server, such as Zope 2, Plone, Grok, and Repoze. This consensus has
driven the splitting up of Zope 3 into a range of independently
released libraries.

To distinguish between these two perspectives on Zope 3, we will
introduce a new term: "the Zope Framework". "Zope 3" should be used to
indicate the Zope 3 application server that is one consumer of these
libraries. To be explicit we encourage the use of "Zope 3 application
server" to indicate Zope 3 and discourage the use of "Zope 3" to
indicate the Zope Framework itself.

The Zope Framework
------------------

What is the Zope Framework? It is a set of libraries that can be used
in the construction of web application frameworks and web application
servers. Use for the web is its primary purpose of the Zope Framework.

Many individual libraries within the Zope Framework can also be used
for a wide range of other purposes unrelated to web development; this
is a secondary purpose of the Zope Framework.

The Zope Framework has a number of important consumers from within the
wider Zope project, such as Zope 2, Zope 3 and Grok. The Zope
Framework project also supports consumers that themselves are not
managed by the Zope project but are nevertheless significant consumers of
Zope technologies. These consumers include projects such as Plone,
Silva and Repoze, and libraries and applications that use Zope
Framework technology.

The Zope Framework has a central status within the wider Zope
project. It is the core Zope technology ingredient to Zope projects
such as Zope 2, Zope 3 and Grok.

Core libraries
--------------

The Zope Framework is a set of libraries. These libraries are released
independently, but typically build on each other.

A library that at some point in time is considered to be part of the
Zope Framework is called a "core library". The Zope Framework contains
those libraries that are reused by a large number of projects, or that
the Zope Framework developers want to promote to be being more widely
adopted. The Zope Framework developers especially favor inclusions of
libraries that are used by other Zope projects.

The set of libraries that is "core" can change over time depending on
how these libraries evolve and are used. New libraries considered to
be "core" can be added to the set, and existing libraries once
considered "core" can be removed from the set.  We should be careful
though, as we cannot just drop libraries from the core without
considerable thought. A library being in the core signals a level of
commitment to this library.

How do we determine which libraries are part of the Zope Framework,
and which libraries are not? The set of Zope Framework libraries is
not static; what is included continuously evolves.

Here are some guidelines on whether something is core or not:

* if it's used widely in our community by the different consumer
   platforms, it's likely core.

* if it's used by only a single consumer platform, it's likely not
   core.

* if only a few people use it, it's likely not core.

* if it has a lot of people who contribute to it from our community,
   it's likely core.

* if it's something we want to encourage more consumer platforms use,
   it's likely core.

* if it contains specific user interface code, it's likely not
   core. If it contains code to help construct user interfaces however,
   it can be core.

These are just guidelines; we can develop new guidelines over time and
adjust these. The final decision on what should be core or not should
be with the Zope Framework Steering Group, as detailed below.

Which libraries should be core to start with? Proposed is to take the
``zope.*`` libraries. We can immediately weed out a lot of libraries
that aren't considered core, and we can then start the slower
processes of adding and removing from that set over time.

Libraries may have a different status in the core to convey extra
information about them, such as deprecation status.

Zope Framework releases
-----------------------

As a service to the users of the Zope Framework, the Zope developers
also make available lists of version numbers of core libraries that
have been tested to work together as a "Known Good Set". This receives
a version number and is the Zope Framework release.  Users of the Zope
framework can use this list, but may also diverge from it where they
see fit. Other projects (such as the Zope 3 application server and the
Grok project) also have a Known Good Sets that expand on the Zope
framework list (and may diverge from it). Each of these consumer
projects can of course have their own release process, schedule and
version numbering.

The KGS concept may be expanded beyond this point to help convey more
information about the libraries such as deprecation status.

Above we described the concept of a Known Good Set for a collection of
libraries (or framework). We have currently technical infrastructure
that was developed to maintain the KGS for the traditional "Zope 3"
system. We will need to adjust the technical infrastructure to also
support a KGS for the "Zope Framework" itself, so we can separate it
from the KGS for the app server. This means the KGS infrastructure
will be usable for more than a single project, and this means other
projects may choose to start using this infrastructure as well.

Status of the ZMI
-----------------

Currently intermixed with the Zope Framework core there is code that
implements a particular user interface, the Zope 3 ZMI. There is a
consensus that these application-like parts should be removed from the
core set, as that code typically only sees use by a single consumer
(the Zope 3 application server). It is not used by other consumers of
the framework.

Extra libraries
---------------

Surrounding the Zope Framework core libraries a large number of other
libraries exist that are developed in association with the Zope
Framework. These libraries integrate with the Zope Framework and make
use of the Zope Framework. These libraries are often maintained by
developers that are also Zope Framework developers, and similar
development practices are used.

We will call these libraries "extra". Libraries in the extra group are
sometimes candidates for inclusion in the core, or might be libraries
formerly part of the core but still being maintained. In general some
development philosophies and practices will be shared between the core
and extra group of libraries.

Examples of "extra" libraries are the "hurry.query" library for
constructing catalog queries, the "z3c.form" related libraries for
form generation, and the "grokcore.component" library which contains a
different way to configure components.

Any library that is developed for integration with the Zope Framework
in the Zope repository can be considered "extra"; "extra" is the set
of libraries that is not in the Zope Framework but can work with it. By
having an "extra" designation we can more easily discuss such libraries.

The Zope Framework Steering Group
---------------------------------

The Zope Framework Steering Group is there to provide leadership for
the development of the Zope Framework. The Zope Framework Steering
Group consists of a small number of developers. The Steering Group
takes on the task to represent the interests of the consumers of the
Zope Framework. The Steering Group searches for consensus in the
community of users and contributors and thereby guide the evolution of
the Zope Framework.

The Steering Group nurtures the Zope Framework community, channeling
energy in the community productively, documenting processes and
documenting the framework itself, and evolving the code base. It tries
to make the processes that are already taking place within the Zope
development community more effective. It also resolves deadlocks: in
case of disagreements within the community about particular
development decisions, the Steering Group can make a decision based on
what it believes is in the best interest of the current and future
users of the Zope Framework.

The Steering Group isn't there to come up with full-fledged plans for
development directions itself, but to provide leadership so that the
community can and will come up with these plans. The Steering Group is
there to act as a catalyst for the community, enabling cooperation,
helping to make good decisions, to have a positive atmosphere, and so
on. The Steering Group should have a role of communication,
relationship building, stewardship, removing obstacles, providing or
making possible to self-provide necessary infrastructure, etc. In
order to do so the Steering Group may have to make technical decisions
that otherwise do not get made in order to make some progress.

The Steering Group is not there to lead the development of the Zope 2
or Zope 3 application servers nor for example Grok; these are merely
important consumers of the framework. The Zope Framework steering
group also does not control the development of the extra libraries in
the repository, except where such a library is considered for adoption
within the Zope Framework itself as a core library.

Policies created by the Steering Group for the development of packages
within the Zope Framework can of course be adopted by the developers
of "extra" packages as well. This does not mean that the Steering
Group is responsible for enforcing these policies for these other
libraries; it only has the mandate to enforce these policies for the
core libraries, and it is not within the mandate of the Steering Group
to police other libraries in the repository.

Steering Group can of course make suggestions for improvements if it
notices a library tries to integrate tightly with the Zope Framework,
and when an extra library is considered for adoption within the Zope
Framework itself the Steering Group *can* request changes.

In order to be agile and not be prone to deadlock in itself, the
Steering Group should be smaller rather than larger. It does not need
to directly represent all of our diverse community interests. It
should have people that the community trusts to take their interests
into account and that are trusted to act effectively. In order to
determine who is in it we could devise an election procedure by Zope
Foundation members.

Summary of concepts
-------------------

* Zope 2: the Zope 2 application server.

* Zope 3 (preferred: Zope 3 application server): Zope 3 as an
   application server, includes a way to create projects. Currently
   also contains the ZMI.

* Grok web framework: Grok, very similar to the Zope 3 application
   server, but with extra Grok libraries and policy, and Grok
   community.

* Repoze: a set of libraries that use Zope technology, reuse Zope
   concepts and expand on Zope technology.

* Plone: a CMS based on Zope 2 and the CMF that is a major consumer of
   Zope Framework technology.

* Zope Framework: the collection of Zope Framework core
   libraries. Shouldn't include the ZMI and doesn't include a
   particular way to create a project.

* Zope Framework release: a set of Zope Framework library versions
   that have been tested to work together. This set receives a
   collective version number ("Zope Framework 3.5"). A release could
   simply consist of a list of version numbers.

* Zope Framework Steering Group: the group responsible for the
   leading Zope Framework development, ensuring its continued evolution
   driven by the concerns of the various consumers of the framework.

* Zope core library: a library within the Zope Framework.

* Zope extra library: a library not within the Zope Framework. Could
   be "just not" within the Zope Framework, or "not yet", or "not
   anymore". These libraries are intended to work with the Zope
   Framework and are maintained by the wider Zope community.

Better naming?
--------------

It may be that the community would be served by a better naming of the
projects specified before. Naming discussions tend to be fraught with
controversy and tend to attract a large number of opinions. Improving
the structure and naming of projects outside of the Zope Framework
project is not a direct goal of the Steering Group, but could be a
goal of the wider Zope project, perhaps through the Zope Foundation.