[Dev] Schema circularity and extensibility problems

[Phillip J. Eby]

During the m5 milestone, there were at least three schema problems that I 
know of, that shared a common root cause:

* AbstractCollection.subscribers - it was desired that this be a 
bidirectional reference attribute, but there was then no meaningful place 
to put the "other end" of the relationship, without creating a dummy 
abstract class like "Subscriber" to act as a mixin.

* AbstractCollection.color - it was desirable for certain parts of the 
system to associate a color with collections, but this required modifying 
the AbstractCollection base class to add the attribute, even though 
collections in the abstract don't have color.  :)

* sharing.UIDMap.items - this was a collection mapping iCal UIDs to 
calendar events, but to be a bidirectional reference, it needed an inverse 
attribute on CalendarEventMixin, thereby creating an undesirable circular 
dependency between osaf.pim.calendar and osaf.sharing.

In addition to these three specific issues, there also have been occasional 
problems with people getting "cannot be modified after use" errors, or 
other errors having to do with setting up bidirectional relationships 
across parcels or between more than two kinds.

The common cause of all of these problems is that there is currently no 
easy way for a parcel to add attributes to existing kinds, without 
modifying the code of the existing kind to refer to the new kind.  This 
problem also affects third-party extenders of Chandler.  For example, 
suppose somebody wants to create an "accessibility" parcel that allows 
assigning sounds to collections instead of colors?  :)


So, here's what I'd like to propose:

1. Allow defining "anonymous" inverse relationships.  If we wanted 
'AbstractCollections.subscribers' to be a bidirectional reference, we would 
need only do this:

     # create an unordered, many-to-many relationship with Item
     subscribers = schema.Many(inverse=schema.Many())

Similarly, the sharing.UIDMap.items attribute could be defined with:

     # create an ordered many-to-one relationship
     items = schema.Sequence(pim.CalendarEventMixin, inverse=schema.One())

2. In the event that something like an attribute editor (or some other 
object or API that needs an attribute name) needs to be pointed at one of 
these "anonymous" attributes, they will be accessible via a "fully 
qualified" attribute name.  For example, to access the collections an item 
is subscribed to, you could get its 
"osaf.pim.AbstractCollection.subscribers.inverse" attribute.  You can't get 
this attribute statically in Python code; you have to use getattr() or 
getAttributeValue() or any of the other normal APIs that take attribute 
names, and pass in the string 
"osaf.pim.AbstractCollection.subscribers.inverse".

3. Implement a convenience API that lets you use the .inverse directly, in 
place of using the long name, e.g. something like this:

     pim.AbstractCollection.subscribers.inverse(someObject)

could perhaps be used to get the attribute in a more type-safe way.  If the 
attribute is frequently used in a given module, it can do something like 
this at the top of the module to create a shortcut:

     subscribees_of = pim.AbstractCollection.subscribers.inverse

and then just use it directly:

     # returns the collections someObject subscribes to
     subscribees_of(someObject)

4. For the case where both ends of a relationship already exist (e.g. 
AbstractCollection and ColorType), we would allow defining the necessary 
attributes as follows:

     class CollectionColor(schema.Relationship):
         collections = schema.Many(pim.AbstractCollection)
         color = schema.One(blocks.ColorType)

Defining this class would create "anonymous" attributes on the relevant 
kind(s).  If one side of the relationship is a type (e.g. ColorType), then 
this would just create an anonymous value attribute on the kind (e.g. a 
"CollectionColor.color" attribute on the AbstractCollection kind).

If both sides of the relationship are kinds, however, then each gets an 
attribute that points to the other, creating a bidirectional reference 
without modifying either kind's class definition.   For example, if a third 
party parcel wanted to create a "likes" relationship between contacts, it 
might do:

     class Likes(schema.Relationship):
         likees = schema.Many(pim.Contact)
         likers = schema.Many(pim.Contact)

This would create a many-to-many relationship between contacts, *without* 
requiring the base Contact type to be modified.  However, it will not 
conflict with any other third-party extension that creates such attributes, 
nor will it be affected if Chandler later adds "likees" and "likers" 
attributes to Contact.  This is because the attribute names created by the 
above code will be "some_parcel.Likes.likees" and 
"some_parcel.Likes.likers", and these names will therefore not conflict 
with a "likees" or "likers" that might be defined by some other parcel.

To navigate this relationship, you would simply use the likes and isLikedBy 
class attributes, as before:

     Likes.likees(somebody)      # get the contacts who somebody likes
     Likes.likers(somebody)      # get the contacts who like somebody
     me in Likes.likees(you)     # do you like me?

     Likes.likees(everybody).add(somebody)  # everybody likes somebody!
     Likes.likees(me).remove(you)           # I don't like you any more

As you can see, this provides a fairly usable API for parcels that create 
new relationships; it's not quite as convenient as being able to say 
'somebody.likees' directly, but it doesn't require the Chandler-supplied 
schema to anticipate every possible future need, and it doesn't create 
circular dependencies between parcels.

Some of you may remember ideas like these from my Spike prototype earlier 
this year, so these are not really anything new.  There's even a documented 
implementation of them in Spike; see:

     http://svn.osafoundation.org/chandler/trunk/internal/Spike/src/spike/schema.txt

and there's some additional discussion in:

     http://svn.osafoundation.org/chandler/trunk/internal/Spike/src/spike/overview.txt

But at the time I was creating the Spike-like schema API for Chandler, it 
was not at all clear to me how I could implement these ideas using the 
repository, and also the need for them didn't seem to be an immediate 
issue.  Now, however, the need has popped up repeatedly, and with Andi's 
help I've figured out a basic idea for how to make more or less the same 
API work atop the repository's schema mechanisms.

I'd like to hear your comments and questions, to make sure this is going in 
the right direction.  By the way, I believe these changes should also allow 
us to get rid of some of the annoying schema errors we currently get that 
result from circular dependencies, including the dreaded "cannot be 
modified after use" error, and we might also be able to get rid of the 
confusing distinction between 'otherName' and 'inverse', as 'otherName' is 
essentially only needed right now as a workaround for the absence of the 
API features I've described here.