[ZWeb] DISCUSS: Wiki subscription slowness

[Kapil Thangavelu]

hi folks,

--- alan runyan <runyaga@runyaga.com> wrote:
> Yes.  I ran into this when I was doing some workflow
> stuff (where it sent
> notifications off) in the CMF.
> I haven't got around to optimizing it but *what I am
> thinking* of as a quite
> asycn. workaround is to use
> kapil's Events Product.  then some cron job could
> scoop up all the emails
> and do what is necessary.
> 
> in Java land we would solve this by putting the mail
> notification on a JMS
> queue.  events is the closest thing
> we have.  there should be a MessagingQueue for
> Python.  but I think Events
> could be the path of least resistence (?)
> or injecting it straight into the queue?


well since the cat is somewhat out of the bag ;) i can
answer about how gideon handles this type of thing.

there are three core components for it Event Channel,
BulkTxnMailer, and Scheduling. and one content level
service Subscriptions.

The Event Channel is roughly analagous to a java style
jms system, it supports a generic publish subscribe
model, with cataloged event queues (although i'm
considering making the event queues sit in an rdbms).
Most of the content level applications send out
notifications on the event channel when something
interesting happens, this is mainly for application
integration, but the subscription system basically
allows users to subscribe to any application generated
event (although the application can 'hide' event
types). the user subscribed events are delivered to an
event queue, so that the potentially thousands of
emails and event listener notifications don't need to
happen for every subscribable event. because of this
the subscription system supports user defined
notification periods on an individual event basis, ie
instead of getting spammed with wiki notifications,
you could elect to get a daily summary.

the scheduling component is a new cron system for
zope, 
which supports non-persistent tasks, and limited
support for zeo operation. while the value of doing
cron functionality within zope is a separate
discussion, i think on the whole it adds to the
platform. the scheduling system has tasks to clear out
the event queue and send mail periodically (via
smtplib)

the bulktxnmailer is a modified mailhost, it employs a
separate product TransactionalFS to give it some
integration with the zope transaction system, it dumps
mail to queue a directory for latter sending by the
scheduled mail delivery task. (there is also a non
bulk  mailer for those who just want
transactionality). of course the email protocol isn't
transactional, so this the txn integration mainly just
consists of delaying the sending till the last
possible moment. this isn't currently part of the
subscription system (which sends directly via smptlib
from the cron task) just for general sending of email
from gideon.

one of the things i've been considering is adding
callbacks to the mail delivery task so that it can
disable a subscription if the mail becomes
undeliverable.

in terms of pure speed, injecting into a mta queue is
the best, there is some c code on zope.org as part of
how to for bulk mailing with postfix and zope. but if
the mail sending is async anyways i'm fine with just
using smtplib, esp. if i can get the benefits of
callbacks to update subscriptions based on the mta
responses.

well, i'm off to look for a job.

cheers

kapil


> 
> ----- Original Message -----
> From: "Paul Everitt" <paul@zope.com>
> To: "Lennart Regebro" <lennart@torped.se>
> Cc: <zope-web@zope.org>
> Sent: Friday, April 05, 2002 10:07 AM
> Subject: Re: [ZWeb] DISCUSS: Wiki subscription
> slowness
> 
> 
> >
> > Your (4) is essentially my (1).  Unfortunately it
> isn't as if we'll have
> > this in the next hour.  Until then?
> >
> > --Paul
> >
> > Lennart Regebro wrote:
> > > From: "Paul Everitt" <paul@zope.com>
> > >
> > >>1) Turn off notification until it can be
> reimplemented.
> > >>
> > >>2) Stop authoring in Wikis.
> > >>
> > >>3) Move the Wiki authoring for new.zope.org
> requirements (nzo) over to
> > >>new.zope.org.
> > >
> > >
> > > 4) Utilizing a proper mail handler queuing. I
> haven't done any
> performance
> > > tests on this, but putting a mail through
> sendmail/qmail-inject/whatever
> is
> > > probably much faster than doing an SMTP
> connection, right? That would
> make
> > > the sending much faster.
> > >
> > > I can make a MailHost that instead of an smtp
> host has the name of a
> queue
> > > injection program (by default sendmail). This
> would mean, however, that
> you
> > > must have a mailserver like sendmail or qmail
> installed on your
> webserver. I
> > > don't know how this would work with ZEO either,
> I guess each server
> needs
> > > the mail server installed?
> > >
> > >
> > >
> > >
> > >
> > > _______________________________________________
> > > Zope-web maillist  -  Zope-web@zope.org
> > > http://lists.zope.org/mailman/listinfo/zope-web
> >
> >
> >
> >
> >
> > _______________________________________________
> > Zope-web maillist  -  Zope-web@zope.org
> > http://lists.zope.org/mailman/listinfo/zope-web
> >
> 


__________________________________________________
Do You Yahoo!?
Yahoo! Tax Center - online filing with TurboTax
http://taxes.yahoo.com/