[Zope-CMF] Project-style workflows, DCWorkflow, SQL, etc.

[Phillip J. Eby]

Kent: I hope you don't mind me cc:ing this to zope-cmf, I think many of my 
comments are applicable there.  Thanks.


At 08:14 PM 9/30/01 -0500, Kent Polk wrote:
>Phillip J. Eby wrote:
>
> > In order to better understand the scope of the project, I have one other
> > question: what does DCWorkflow lack?  That is, what keeps you from doing
> > everything you need with DCWorkflow?  Based on the information you've sent
> > me, I don't see what stops you from using DCWorkflow, unless it's due to:
> >
> > 1) one or more non-functional requirements, such as clarity/maintaibility
> > of code, or the ability of non-programmers to develop complex workflows
>
>Bingo. Several of us have been working on how to do this, but
>
>a) We (CMF list, etc) don't really know how to create a 'Project
>Workflow' kit. We don't really know how to describe the functional
>requirements. I know what *I* need for my tasks, but I don't really
>know what a Project Managerment kit ought to really look like.

Neither do I, although I have some ideas.  Ty and I came up with a concept 
once that was based on the notion of "waiters for".  That is, the 
fundamental notion was dependency.  A task could be "waiting for" a person 
or persons to do it.  It might be "waiting for" some set of 
subtasks.  Things "waiting for" persons or groups of persons show up in 
their worklists.  Things "waiting for" automated processing of some kind go 
in a processing queue.  Someone who makes a request can also be said to be 
"waiting for" the request, although more often than not, they will actually 
be making a request in the context of some other task, which will then be 
"waiting for" the requested task.

Waited-for items send messages to their waiters, which may cause the waiter 
to stop waiting, or perform other actions (such as consider itself 
completed).  In the case of a task performed by a person, they "send a 
message" by using web forms/methods on the task.  In the case of 
task-to-task dependencies, the "Observer" protocol could be used, with the 
waiter subscribing to the waited-for item's events.

In this model, a "request" is just another waiter/waitee, like a task.  The 
main difference is that a request has a generic workflow - that of 
responding to a request to do something.  The receiver of a request (who is 
what the request is "waiting for") then creates dependencies from the 
request to the tasks that are needed to fulfill the request.  In a more 
advanced system, specialized request types could make creation of the 
standard tasks automatic - although in that case it may make sense to 
simply customize the workflow and make it purely a task rather than a 
request.  In general, it has seemed to us, however, that there is value in 
distinguishing the tracking of a request to do something, from the actual 
doing of it.

Anyway, this model is very simple and requires only two basic interfaces; 
the vast majority of objects in the system will implement both.  People, 
tasks, requests, etc. are all both waiters/waitees, requesters/responders, 
dependents/dependencies, or whatever you want to call them.  Most of the 
objects can be considered state-machines which go through a set of 
transitions ala DCWorkflow.  Events on waited-for objects can trigger 
transitions in their waiters, which may use guard conditions that check 
whether there are any objects still being waited-for.

Although this model is simple, implementations can be arbitrarily 
complex.  For example, you may want people or "worker" objects to be able 
to do things like send e-mail notifications (or page/instant-message them) 
when something is added to the person object's list of objects "waiting 
for" them.

DCWorkflow is a bit primitive in some of these areas, but there's nothing 
in the above model that's inherently beyond its capabilities.  Your 
principal difficulties would be in making it clear how these things 
interact with one another, and perhaps creating some libraries or base 
classes for dependency management, so that the guards, etc. in your 
DCWorkflows are not too complicated to write.

Also, one other potential complication would be in the area of users; 
you'll likely need to extend the notion of a Member or create some kind of 
"worklist" object that would act as a proxy for the member and implement 
the "waitee" interface.



>We think that the CMF and DCWorkflow has the bulk of the functionality
>there, but a lot of filler pieces, etc are missing.
>
>b) We don't know how to describe how the components of a project
>workflow kit ought to interact with the Workflow.
>
>  What is the workflow manager? Is it an object that interacts with
>  it's workflow, the objects that are created, and their workflows
>  or is it a workflow itself?
>
>  How are the workflow objects to be stored and managed? Should they
>  be children of some parent object or should they be gathered
>  together by some query mechanism, regardless of where they reside.
>  An example of this could be that workflow objects are created and
>  stored in their owner's folders, but the display/management
>  interface locates them, etc. Very important design issues, as far
>  as I can see.

These need not be global decisions.  All that the "waiter" concept requires 
is that a waiter be able to find and link to an existing "waitee" or create 
a new one.  In the vast majority of use cases, however, one creates a new 
waitee, or, if the waitee is a person/group, link to an existing one.  It 
tends to be a fairly infrequent action where one needs to create a 
dependency to an already existing task, so such use cases can usually be 
addressed on a case by case basis.  For example, in the case of an ISP, 
they may receive many requests from customers regarding a network outage, 
and so they might need to make all these requests "wait for" a single NOC 
ticket, in which case their requests need to have the ability to find and 
link to NOC tickets.  But then you know in your specific design where to 
look for NOC tickets.  Make sense?

Thus, at a framework level, it suffices to have a registry (such as the 
existing Zope/CMF type registries) for selecting types and creating new 
waiters, and a way to look up users or worklists (such as some extension to 
the Membership machinery), and perhaps a standard way to find 
"Organization" or "Team" objects (for department, role, or skill-based 
worklists, as opposed to specific individuals).


>I want to end up with a product which, by default, performs a
>generic workflow, using std CMF item types (plus a generic 'request'
>item type) that can be adapted to work with more complex object
>than the current CMF item types.
>
>c) We don't know how to adapt DCWorkflow so that it can adjust
>somewhat to human interaction. Right now, it's pretty static, though
>people are working on how to extend it to provide this sort of
>functionality.
>
>d) I also don't really trust ZODB with my data. I want it packaged
>through SQL or something more reliabe/reconstructable. This starts
>getting complicated pretty quickly.

Yes, if you plan to work with CMF, this is harder.  The CMF workflow 
protocols are plenty abstract enough to allow for this, but DCWorkflow 
itself is not.  You could possibly subclass it, however.

Another possibility is ZPatterns, but I wouldn't recommend it, due to its 
lack of a future.  Steve Alexander has been making heroic efforts to keep 
it going and even move it forward, but I personally think that the 
NewReligion will make it ultimately obsolete.

Regardless of your workflow implementation or even specific DB software, I 
do think that the WarpCORE design patterns will make your SQL design 
easier; specifically, it's geared to polymorphism, hierarchies, and 
references to external objects (e.g. data stored in those "evil scientific 
software" apps you mentioned previously).



> > 3) you're not clear on how to map these interacting workflows into
> > DCWorkflow's model.
>
>Exactly! :^) That's my problem. I can't map my problem onto
>DCWorkflow. I have many ideas about how one might interact with a
>project manager, but I have a major disconnect between that and
>the code I've seen in different workflow managers. This is the key
>problem that I need help with.
>
>I can describe what I need, but I don't know how to map it in any
>reasonable fashion onto anything that I've run across. That's my
>limitation and I know it. That's primarily what I need help with.
>
>(I don't have anyone around here to discuss this with that understands
>anywhere as much as I do and I'm woefully inadequate)
>
>Talking with Ty in the past, I really suspect that you have some
>good ideas and experience about how to build a Project Management
>workflow kit. That's what I want to tap into. I know what needs to
>be done with things specific to SFBR, but not how to map that onto
>some nebulous workflow manager. It needs to be quite a bit less
>nebulous before I can start to see how it might be done.
>
>Basically it's too far over my horizon. I want someone to help me
>build a tower so I can see further, and I don't know how to engineer
>a tower - or even what's important when you engineer a tower.
>
>Thanks!

Does the waiter/waitee model make things any clearer?  It's possible to 
further extend it by adding extra info to the "waiting for" links, such as 
deadlines, priorities, began/ended dates, etc., but even without these 
things, the basic model offers quite a lot.