[Chandler-dev] Office calendar out of whack

[Morgen Sagen]

This is complicated, so splitting into two sections:

Executive summary

The migration path from cosmo-demo to osaf.us has hit a snag,  
exposing a problem with keeping events in sync.  We have a mix of old  
Chandlers and new, old data and new, and this is causing problems.   
Event-related attributes are being shared properly, but any non-event  
data like stamping may only sync with some other Chandlers.  Also,  
this mix of old and new has caused around 270 extra XML files to be  
published, and it's causing each sync to be slower than it should  
be.  Below is a deeper explanation, and a proposal to fix it.   
Basically, continue to use the osaf.us office_calendar-1 that Sheila  
mailed out (knowing that only event related attributes are being  
properly shared).  When I have a fix ready I will ask everyone to  
export any calendars they wish to keep to .ics files, install a new  
Chandler, import the .ics files, and publish to new osaf.us collections.


Gory details

The migration path from cosmo-demo to osaf.us has hit a snag,  
exposing a problem with keeping events in sync.  It has to do with  
what happens when we import a .ics file: in the past (more than a  
couple weeks ago) we generated random UUIDs for each event,  
independent of each event's icalUID.  So the first Chandler to import  
a .ics file gets to assign each event item's UUID, and as long as  
that event was shared via Cosmo, the UUIDs were kept in sync.

As of a couple weeks ago, we improved this by trying to convert the  
icalUIDs in an imported .ics file directly to item UUIDs, so that two  
Chandlers independently importing the same .ics file would assign the  
same UUIDs to those events.  Therefore, even if two Chandlers happen  
import an .ics file, when they shared those items, the items would  
correlate since the UUIDs match.

What happened during the migration of the office calendar from cosmo- 
demo to osaf.us was instead of subscribing to the cosmo-demo version,  
unsubscribing, and republishing to osaf.us, the office calendar was  
imported again via .ics -- thereby assigning all new UUIDs to those  
events -- and then the calendar was published to osaf.us.  If all the  
dog-fooders who then subscribed to the new calendar had started with  
clean repositories, we actually would have been ok.

However, at least one Chandler still had events from the office  
calendar on cosmo-demo in their repository, and when they subscribed  
to the osaf.us office calendar, they got new versions of the event  
items they already had -- but with different UUIDs.  The problem was  
then exacerbated when they later synced, and started publishing their  
notion of what the UUIDs should be to the server, resulting in two  
parallel sets of event items.

Depending on whether you had cosmo-demo era versions of these events  
in your repository, or whether you first subscribed *before* the  
parallel set of events got published to osaf.us, your Chandler can be  
honoring either one set or the other.  This actually half works:  
since the icalUIDs still match, any event-related attributes actually  
*do* sync up -- things such as start time, duration, recurrence,  
title.  However, any data transmitted via XML such as stamping and  
non-event attributes would only be kept in sync with Chandlers  
honoring the same set of items.  So there is a 50/50 chance someone  
else would see you stamped an item.

Oh yeah, this problem is further exacerbated by people running old  
versions of Chandler, which don't have various bug fixes.

So what to do?

We first need to be able to detect when Chandler is trying to import  
events they already have under a different UUID.  This can't be  
allowed, so I propose adding logic to detect this during subscribe/ 
sync, aborting the sync if it happens.  If it's possible to somehow  
reconcile the events I will do that, but I have to think that through  
some more.

Next we need to guarantee that an old Chandler doesn't come in and  
mess things up, so I will bump up the XML version number.  This means  
old Chandlers won't be able to sync new shares, and new Chandlers  
won't be able to sync old shares.  An unfortunate outcome, but we  
really have a mish-mash of data out there (events whose UUIDs have no  
correspondence to their icalUIDs), and we need to start "clean".

Third I will add logic to the ICalendar import code to  
deterministically compute an event item's UUID from its icalUID  
(thanks to Jeffrey for this idea) using an md5 hash (only if the  
icalUID is not already in a UUID friendly format):

    import md5
    from chandlerdb.util.c import UUID

    try:
       uuid = UUID(icalUID)
    except ValueError:
       # icalUID not in UUID friendly format
       uuid = UUID(md5.new(icalUID).digest())

Really, the only reason we need this third step is to support  
Chandler trying import a .ics file that comes from a client who  
doesn't use UUID friendly icalUIDs (perhaps Outlook).

When I have these fixes ready I will ask everyone to export any  
calendars they wish to keep to .ics files, install a new Chandler,  
import the .ics files, and publish to new osaf.us collections.

~morgen
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.osafoundation.org/pipermail/chandler-dev/attachments/20061109/becba935/attachment.htm