[Dev] The Chandler/Cosmo sharing format

[Morgen Sagen]

I want to get people's opinions on the format Chandler should use  
when publishing items to Cosmo.  I'll begin with an overview of the  
current state of Chandler/Cosmo sharing:

When Chandler publishes an item collection, it creates a CalDAV  
calendar collection (which contains .ics resources for each  
CalendarEvent in the item collection) and a ".chandler" subcollection  
(which contains .xml resources for all items in the item collection,  
even the CalendarEvents).  The .xml resources are in our own flavor  
we refer to as "cloud xml", and it's tied closely to our internal  
domain model (PIM schema).  Each .xml resource contains the shared  
item's kind (actually its python class), its UUID, and each shared  
attribute/value pair.  If there are other items in the main item's  
"sharing cloud", then those items are included in the XML too.  In  
the case of CalendarEvents, we simply leave out the attributes that  
are represented in the iCalendar resource, so we don't duplicate data.

Here's an example of the current "cloud xml" which defines a  
CalendarEvent and its organizer (a Contact item in the event's  
sharing cloud):

<?xml version="1.0" encoding="UTF-8"?>

<CalendarEvent version='2'  
class='osaf.pim.calendar.Calendar.CalendarEvent'  
uuid='caaa1210-6cd9-11da-9e5b-000a95bb2738'>
    <icalUID>caaa1210-6cd9-11da-9e5b-000a95bb2738</icalUID>
    <organizer>
       <Contact class='osaf.pim.contacts.Contact'  
uuid='aa79fff0-6cd9-11da-9e5b-000a95bb2738'>
          <emailAddress></emailAddress>
          <contactName>
             <ContactName class='osaf.pim.contacts.ContactName'  
uuid='aa798c14-6cd9-11da-9e5b-000a95bb2738'>
                <firstName>Chandler</firstName>
                <lastName>User</lastName>
                <createdOn>2005-12-14 11:41:48.467422</createdOn>
             </ContactName>
          </contactName>
          <displayName>Me</displayName>
          <createdOn>2005-12-14 11:41:48.471043</createdOn>
       </Contact>
    </organizer>
    <displayName>New Event</displayName>
    <createdOn>2005-12-14 11:42:42.480434</createdOn>
</CalendarEvent>


The benefits to what we have now are:

1) We have a general purpose method for sharing *anything* between  
Chandlers, including binary Lobs (we can share Photos today for  
example); serializing an item is simply determining which attributes  
to share (based on sharing clouds), and stepping through them one by  
one, serializing each value.  References to items in the cloud are  
handled by recursion.
2) It's not much work to specify which attributes should be shared  
when a new kind is defined (you just add those attributes to the  
kind's sharing cloud)
3) The UUID of a shared item is the same on different Chandlers  
sharing that item
4) We can still interoperate with other CalDAV clients, since we  
import/export .ics resources
5) It's easy to support stamped items; the attributes from the  
various mixed-in kinds can all be included in the XML

The drawbacks are:

1) Since the sharing format is tied to our schema, two different  
releases of Chandler won't be able to share the same collection if  
the schema changes in an incompatible way
2) We have to publish two different resources to share a single  
CalendarEvent (or any item that is stamped as an event); this causes  
problems when, for whatever reason, there is a .xml resource on Cosmo  
without a matching .ics resource
3) Publishing two different resources for each item takes twice as  
long :-)
4) We're not using 'standard' formats for kinds other than CalendarEvent


So let's set out some goals for the sharing format...

1) The format should not be so tied to our domain model as to be  
changing with every Chandler release.  The format should be 'long  
lived' like iCalendar.  This would allow different Chandler versions  
to share a collection.

2) The format should be extensible to support new data types

3) We should strive to use existing standards.  Some obvious ones:
    - test/vcard			vCard v3.0		http://www.ietf.org/rfc/rfc2426.txt
    - text/calendar		iCalendar		http://www.ietf.org/rfc/rfc2445.txt
    - text/message		MIME Message	http://www.ietf.org/rfc/rfc2045.txt

4) The format should support our notion of stamping (where an item  
can of multiple types simultaneously)

5) We shouldn't have to publish multiple resources to share a single  
item

Issues with these goals:

1) We need to decide on some formats and stick with them long term.   
Our domain model is still in a bit of flux, so too might the sharing  
format(s).  Hopefully the schema will settle down in 0.7.

2) We trade off the general-purpose import/export conversion for kind- 
specific converters that would have to be written, just like Jeffrey  
did with ICalendarFormat+vobject; however, in some cases this is  
trivial like for email since we already persist the raw MIME message

3) If we want to share an attribute that doesn't fit in with these  
standards, what do we do?  We solved this by using our own XML  
schema, and I suppose we could continue to do this, for attributes  
which don't fit a standard.

4) We'll need to be able to store multiple of these 'standard'  
representations within a single DAV resource, which means we would  
need to define some metadata which 'wraps'  the kind-specific  
representations, something like:

<Item uuid="56202724-7e20-11da-89f7-c9a551df6980">
    <Data mimetype="text/calendar">
	BEGIN:VEVENT
	DTSTART;TZID=US/Pacific:20050913T073000
	STATUS:CONFIRMED
	SUMMARY:New Event
	UID:fe033476-249e-11da-d1a9-000a95bb2738
	DTEND;TZID=US/Pacific:20050913T190000
	END:VEVENT
    </Data>
    <Data mimetype="text/vcard">
	BEGIN:VCARD
	FN:Sagen;Morgen
	N:Morgen Sagen
	TEL;TYPE=WORK;VOICE:+1-415-555-1212
	END:VCARD
    </Data>
</Item>

Now that I've typed this, I wonder if it really is worthwhile to use  
'standard' formats if we have to wrap them in this way.  No client  
besides Chandler will understand this anyway.  There is SyncML, which  
does something like this, but I don't know that it supports multiple  
<Data> bodies per <Item> like we require.

5) If we go with CalDAV, I can't find a way around this problem.  I  
thought maybe we could get by with sticking all non-CalendarEvent  
attributes into DAV properties, but CalDAV doesn't let me PUT non-ics  
resources in a calendar collection, so we'd only be able to share  
CalendarEvents.  We could just do calendar sharing and that would  
make things easier, but that's not very cool.  :-)


So I'm torn on how to proceed with this.  On one hand, using standard  
formats is good, and we have the most important one (iCalendar)  
covered.  On the other hand, there may not be a big benefit in trying  
to also use 'standards' for the other data types since we would have  
to wrap them in non-standard ways (bundling them up within a DAV  
resource).  Clearly we need to decouple the sharing format from the  
domain model so that different Chandler versions can share the same  
Cosmo collection, and that is one of my 0.7 goals.

Thoughts?   Solutions I've overlooked?

Thanks,
~morgen