On Thu, 2008-12-18 at 13:00 -0200, Tim Cook wrote:

This issue is such a huge frustration for me that I am offering a bounty of 100USD out of my personal pocket to the first person that solves the issue, gets it committed to a published zope.schema egg and included in the standard Grok distribution.

It seems to me to be a reasonable (though not extravagant) amount since most of the trouble shooting has already been done.

Hi All,

I want to change the conditions of this offer.

Over the past several days there have been too many people to mention help me both publicly and privately with this issue in various ways. Mostly in patience and education.

I would like to donate this small amount to the Python Foundation in the name of all professional Zope developers.

Assuming no large outcry in the next few days this is where the 100 bucks will go.

Thanks, Tim

Hi Dan,

Thanks for taking a look.

On Thu, 2008-12-18 at 20:06 +0300, Dan Korostelev wrote:

Hi, Tim.

I builded out your application and took a quick look at the error. It looks like you're mis-using the zope.schema.Field class and its interface. It should be only used in schema definition, while you are using Field instances as attributes for other objects. For example, you are defining the ObjectId as a field, but are using its instances as objects. That's not a valid use.

Okay. I can accept that it might be a mis-use.

I extracted the problematic code to separate file and changed the IField to Interface and Field to ``object`` (though you'll want to use Persistent, i guess) for ObjectId and ObjectRef and it works okay. :-)

This is actually what I did at first. You may want to refer to the UML http://www.openehr.org/svn/specification/TAGS/Release-1.0.1/publishing/archi...

(This is the rm.support.identification package we are looking at)

Since ObjectId is an abstract class so inheriting from object seems reasonable. However, most of the other id classes inherit from ObjectId and when they are used as attributes to other classes they need to have the meta data expected by zope.schema. Otherwise you get errors like "keyword required not found" (or something similar). This is true throughout the model. So my solution (and CERTAINLY there may be a better one) was to use the Field class (it is just a Python class - right?) as the base class for all of the base classes in the model.

As I said before I may have miss-diagnosed the problem and may fix may break other things?

Or, it might be that I don't fully understand your code and application architecture, so I would like to hear more about it. It looks quite over-engeneered to me.

I fully understand why you might think that.

IF you'll bear with me... healthcare information is very complex (certainly the complex domain I've run into in 32 years of dealing with information systems) and the knowledge domain is constantly changing. I have a short point paper here http://timothywayne.cook.googlepages.com/context-lies.pdf that gives a 30,000' view of the issues of semantic interoperability and computability of healthcare information.

and a good introduction to the domain issues are here: http://www.openehr.org/shared-resources/getting_started/openehr_primer.html

This model has more than 20 years of constantly improving R&D on this subject. The key goal have always been to be abstract enough so that it can be implemented in any OO language on any platform in any type of healthcare setting and all stored patient information is guaranteed to be computable (decision support, etc) as well as maintain it's semantic integrity over the course of time irregardless of the changes in the science. So in essence you can go back to any point in time and not only know what the blood pressure of the patient was, but also "what was known" about blood pressure at that time.

This model has been implemented in Eiffel, Java, C# and VB. The applications (both commercial and open source) have demonstrated these capabilities.

Certainly the model isn't perfect but there is a completely open change management system in place that deals with any changes due to new ares or new science.

In order for this Python implementation to remain as true as possible to the model as well as exercise the value in the ZCA; this is the path I have taken.

Thanks for your help.

Cheers, Tim

Hi Chris,

On Fri, 2008-12-19 at 10:06 +0000, Chris Withers wrote:

Tim Cook wrote:

...

As I said before I may have miss-diagnosed the problem and may fix may break other things?

This is what a full-coverage unit and functional test suit is for. You have got automated tests for all this stuff, right?

NO! I do not! I am a ZCA user not a full time python programmer. That is why I a asking for help here. Initially at least on the Zope Users list.

If you are expecting ZCA users to be full time Python gurus then why have two mailing lists? It seems that you are asking a "user" to provide the "developers" with a ready to commit patch. I wish I had those skills and could do so.

Alas, it isn't going to happen.

Cheers, Tim

Thanks Dieter,

On Fri, 2008-12-19 at 19:41 +0100, Dieter Maurer wrote:

I often approach situations like this with so called "Monkey Patches": I replace or enhance classes or methods during startup (triggered by some startup event).

This may not be optimal but allows me to solve my problems without tight coupling with the core developpers who usually do not have my problems and tend not see them as very high priority.

I am starting to see why other Python developers aren't too interested in Zope. Despite the awesome power of the architecture.

I've been poking around and using and supporting Zope since 1999. But because I am not an expert programmer (my skills are in another field) and because I do not know Zope inside and out. I am being chastised for not supplying my own complete fixes to the developers or at least being able to cover for their mistakes inside my application. Silly me. Working in open source for 14 years and I had no idea it worked like this.

I'm sure glad that we do not run openEHR like that.

Cheers, Tim

Hedley Roos wrote:

Chris mentioned unit tests. You do not have to write a new unit test. What is required is to have a look at the tests and then identify one that is relevant to your problematic method. This test should be very easy to read. The "hard" part is for you to expand this test to demonstrate the exception. This will typically require a few lines of code but since you already fixed a bug I assume it won't be hard to do.

Actually, I disagree with you here... big bloated tests that test huge amounts in one go is not the way to go. I'd suggest that if it's a specific problem, a new test should be added that does the absolute minimum to demonstrate the bug...

Then you have to run all the other tests to check that you haven't broken anything else by doing ./bin/instance test -s zope.something.

Finally create a patch file and attach it to your original report on the tracker.

Mmm, after writing all this I can see why it is a pain.

I can't. The testrunner is already there and buildout-based development makes it trivial to run. svn already does the patch creation for you:

$ svn diff

be able to get past that bit. Perhaps a kind Zope dev will pick up on this.

Unlikely with his attitude...

Chris

Hi there,

Tim Cook wrote:

On Fri, 2009-01-09 at 11:05 +0000, Chris Withers wrote:

[snip]

So again, referring to the subject line. What are the next steps?

I am trying to understand the history of the discussion, which has a lot of text and is widespread.

I think Dan Korostelev tried building out the 'ostep' application and found the error, correct? I just did the same...He then had more patience than I did and tried to understand your source code.

Okay, I'll take another look then and look at ObjectRef. Ah, yes, Dan pointed out to you that you are using a zope.schema.Field in a class instead of in a schema (an interface). That isn't right, and since the direct use of that causes an error, that looks suspicious. Whether it is the cause of the bug or not I do not know.

Generally to help people without patience (like everybody), you should create the absolute *minimum* example of code that demonstrates your bug. This so that people aren't distracted by a vast codebase when they try to debug your issue. I feel a bit guilty this is all taking so long, but I'll also point out this is not the first time I asked for this; I asked the same december 18. We will need such a minimum example for a number of reasons:

* I have no idea how your application is put together and I don't have the time to figure it out. A small application is a way easier conversation piece.

* When you construct a minimum example you will have an easier way thinking about this bug *yourself*.

* If we confirm it is a bug, we will go into zope.schema and add a test case that demonstrates it first, before we actually fix it. This way we will know our fix works (when the test case stops failing for us), and in the future we won't accidentally reintroduce the bug due to changes in the code. You can do part of the work for us. This is not a task for core developers; you are the only one with the knowledge necessary to construct this minimal test case as you understand your application while we don't.

In summary, you have to make the life of the bug fixes as easy as possible. I think that is best done by providing the minimum information necessary for anyone to reproduce and examine the bug, in isolation.

Regards,

Martijn

Hey,

On Fri, Jan 16, 2009 at 11:37 AM, Tim Cook timothywayne.cook@gmail.com wrote:

On Mon, 2009-01-12 at 22:05 +0100, Martijn Faassen wrote:

...

Okay, I'll take another look then and look at ObjectRef. Ah, yes, Dan pointed out to you that you are using a zope.schema.Field in a class instead of in a schema (an interface). That isn't right, and since the direct use of that causes an error, that looks suspicious. Whether it is the cause of the bug or not I do not know.

Thanks to all for the help. I will put together a wiki page that chronicles and explains the entire issue and process of getting here. Along with the simplest example I can come up with.

I don't think a wiki page with a chronicle is necessary or even helpful; you need to give us the information necessary to find the bug, but no distracting surrounding information. To debug this problem, a developer will need the smallest possible example of code that demonstrates the problem. That means, I take it, just 2 schemas and a single form. Describe briefly what you expect to happen and what in fact happens. If that example can be done *without* inheriting from Field that'd be good, as it is true that Field is only to be used inside a schema definition and once someone sees that we'll conclude that's the cause of the problem even though it might not be.

Once we have the example someone can either debug the problem, or tell you what you're trying to do isn't the right way to do it.

Regards,

Martijn

Hey,

...

To debug this problem, a developer will need the smallest possible example of code that demonstrates the problem. That means, I take it, just 2 schemas and a single form. Describe briefly what you expect to happen and what in fact happens. If that example can be done *without* inheriting from Field that'd be good, as it is true that Field is only to be used inside a schema definition and once someone sees that we'll conclude that's the cause of the problem even though it might not be.

It is interesting that in table 4.1 of Philipp W's book it specifically states that Field is the base class for all other fields. So how does one build fields that are noot part of the standard zope.schema?

Yes, you do create new schema fields by subclassing from Field.

It's just that we saw you putting a field not in a schema but in what looked like a concrete object. Perhaps we were wrong in reading your code, and this is one reason why you should come up with a minimum example that demonstrates the problem and only that, without a lot of distracting code surrounding it. You're the best suited person to actually create a minimum example.

Regards,

Martijn

Hi Tim,

Tim Cook schrieb:

Thanks for all the assistance.

On Fri, 2009-01-16 at 18:05 +0100, Martijn Faassen wrote:

...

Yes, you do create new schema fields by subclassing from Field.

It's just that we saw you putting a field not in a schema but in what looked like a concrete object.

This has given me a BIG pause while I'm working on a simpler example. It may actually solve the problem.

Are you saying that in order to create a Field that can be used as an attribute of another class; I should define it in an interface and ONLY in an interface?

Such like pseudo:

import Field class IAbc(Interface)

myNewField = Field( ........

and then when I need to use it in a class, simply state that that class implements(IAbc)?

that's what Dan Korostelev said early in the thread: Use the Field only in a schema definition. You define your schema in an interface class. The interface class describes the interface (fields, attributes, methods) that a class implements.

from zope.interface import Interface, implements from zope.schema import TextLine

class IMyTitleSchema(Interface):

title = TextLine(...)

class MyTitleClass(object or some other baseclass):

implements(IMyTitleSchema)

title = u""

The interface tells other components, e.g. forms: An object instanciated from MyTitleClass has an attribute 'title', and title is a TextLine. So a form can render the correct widget. Other components do other things with this information, like validation when the title attribute is written.

You never use a schema field in something other than an interface class. Don't do:

class MyTitleClass(...):

...

title = TextLine(...)

obj = MyTitleClass()

Sure you can have specialized fields that subclass from Field, TextLine, or another base class. E.g. RegistrationNumber(TextLine) that takes care to validate the input for a special format. But you use them in an interface class, not the class that implements the interface.

..Carsten

Hi Tim.

Unfortunately I didn't follow the discussion lately, so may be the problem is no more, but...

I just committed a fix for zope.schema's ValidationError that makes its repr output more sensible. I'd like community to review those changes and say if they're okay, because changing exception formatting syntax will affect doctest so they should be adapted to new style. But I think it's better to adapt the doctests than to make debugging difficulties for our users, right? :)

About your problem, here's the minimal failing example.

from zope.interface import Interface, implements from zope.schema import Field, Object, Int from zope.schema.interfaces import IField from zope.schema.fieldproperty import FieldProperty

class IObjectId(IField):

value = Int()

class ObjectId(Field):

implements(IObjectId)

value = FieldProperty(IObjectId['value'])

class IObjectRef(Interface):

oid = Object(IObjectId)

class ObjectRef(object):

implements(IField)

oid = FieldProperty(IObjectRef['oid'])

oid = ObjectId() oid.value = 3

ref = ObjectRef() ref.oid = oid

Basically, it fails validation because the IField interface defines some attributes (like default and missing_value) that are not set in the Field base class and neither in ObjectId. So the exceptions are really RequiredMissing.

But the general problem is still that you are misusing Field class as base for your not-fieldy class. It defines specific functionality for describing interface schemas and can be treated specially by other components. If all you need is its "title" and "description" fields, just define a base interface and implementation like:

class IItem(Interface):

title = TextLine(title=u'Title', required=True) description = Text(title=u'Description', required=False)

class Item(Persistent):

implements(IItem)

title = FieldProperty(IItem['title']) description = FieldProperty(IItem['description'])

...and use them as base for your objects.

Hope this helps.

Hi Dan,

On Sat, 2009-01-17 at 01:28 +0300, Dan Korostelev wrote:

Hi Tim.

Unfortunately I didn't follow the discussion lately, so may be the problem is no more, but...

There has been a tremendous amount of help from folks like you. However there is still not a solution.

I have been asked several times for a 15 minute overview. This is tough given the complexity but allow me to ask the question at a more basic level. I believe it is similar to the way I asked it last year, but here goes.

I'm not going to address Field or Object here, just explain the basics.

class DataStructure(Persistence): """abstract class"""

class ItemStructure(DataStructure): """abstract class"""

class ItemList(ItemStructure): u""" Logical list data structure, where each item has a value and can be referred to by a name and a positional index in the list. The list may be empty. """

items = List( value_type=Object(schema=IElement), title=_(u"items"), description=_(u"""Physical representation of the list."""), required=False )

class ItemTable(ItemStructure): u""" Logical relational database style table data structure, in which columns are named and ordered with respect to each other. Implemented using Cluster-per-row encoding. Each row Cluster must have an identical number of Elements, each of which in turn must have identical names and value types in the corresponding postions in each row. Some columns may be designated 'key' columns, containing key data for each row, in the manner of relational tables. This allows row-naming, where each row represents a body site, a blood antigen etc. All values in a column have the same data type. Used to represent any data which is logically a table of values, such as blood pressure, most protocols, many blood tests etc. Not used for time-based data, which should be represented with the temporal class HISTORY.. The table may be empty. """

class ItemSingle(ItemStructure): u""" Logical single value data structure. Used to represent any data which is logically a single value, such as a person's height or weight. """

************************************************* There are others and I left out the attributes and methods of these classes; with the exception of ItemList attribute 'items', where it is a zope.schema List but the value types are restricted to the schema described by IElement (also part of openEHR); but I think you get the idea.

These classes are used as the base software for all openEHR applications. Of course the classes get more complex and deal directly with healthcare related issues.

Now there are thousands of applications that will have data instance with attributes expressed in classes based on the above software that represent single (but complete) clinical concepts. But not all application user interfaces will use all available attributes of the concept. One may use an ItemTable and another an ItemList but they will both be valid in ANY application because the attribute represents a legal instance of ItemStructure.

When that data instance is sent from application to another the receiving applications still needs to know the complete semantic context of when that data was collected. Think medico-legal, research and decision support over the lifetimes of patients and populations. So even if the user didn't see all the options in their GUI; it is still all contained in the data that was transfered.

So how do I build my schemas so that Zope does the validation of nested schemas and even lets me use standard widgets?

I hope this was less than 15 min. For those that want specific examples I can list a few.

Cheers, Tim

Tim Cook wrote:

I DID submit a patch suggestion in the Launchpad bug report.

I think you're talking about this:

https://bugs.launchpad.net/zope3/+bug/301226

Sorry, but the patch doesn't make any sense. Your version of _validate_fields quietly skips validation entirely by default. Why would the function be called "_validate_fields" if it doesn't actually validate unless the caller passes some special parameter? The function needs to validate the fields regardless of the parameters.

I would also like for you to explain just what it is about my "attitude" that you find so offensive/problematic?

Please ignore comments like that. They are not worth dealing with.

I traced down and described the problem to the function level. I suggested a fix and asked for experts to review and apply it. I do not have (and am not qualified to have) commit access to ZCA source.

So again, referring to the subject line. What are the next steps?

Here's a tip: for a programmer, good error messages are like gold. The right error message will save you countless hours (or months) of frustration. Therefore, making error messages readable is one of your top priorities.

You have been presented with a very poor error message. Fixing that error message doesn't look very difficult on the surface, so your first priority should be to make that error message readable. Marius was on the right track:

http://mail.zope.org/pipermail/zope3-users/2008-October/008218.html

Look at the __repr__ method of the ValidationError class in the _bootstrapinterfaces.py module of the zope.schema package. This method was designed to generate short error messages, but in your case it appears to be truncating the error messages altogether. I would start by changing that particular __repr__() method to:

def __repr__(self): return '%s(%s)' % (self.__class__.__name__, repr(self.args))

This version prefers verbosity. At a minimum, I hope this version of __repr__ will change the bizarre message "zope.schema.interfaces.WrongContainedType: [, []]" into something legible.

Still, I don't know if my version of __repr__() is appropriate for everyone else, so I am not going to touch SVN yet. For now, this is just something that might help you find the real cause of the bug.

Shane

Tim Cook wrote:

I would also like for you to explain just what it is about my "attitude" that you find so offensive/problematic?

As a general rule of thumb, anyone who posts with more than on exlamation mark is likely on the wrong track.

Cross posting to several lists is also a bit of a no-no.

Jumping up and down because someone won't look at your edge case problem unless you're offering that someone bucket loads of cash likewise ;-)

Chris