[Chandler-dev] Sharing pain points

[Jeffrey Harris]

Hi Folks,

Last week Mimi and I sat down and talked about pain points associated 
with sharing once collections are set up and working.  We came up with a 
list of issues and possible remedies.

The executive summary is that there are still several identified issues 
causing sharing pain, broadly grouped into:

1. Failure to sync
2. Inaccurate pop-to-now
3. Pop-to-now with the byline showing the wrong editor of the item
4. Items have conflicts, but the conflicts are in error or not obviously 
associated with a user-visible attribute

We came up with possible remedies to these problems, based on my 
understanding of the underlying causes.  It's possible that fixing just 
the first few problems may get the frequency of pain low enough that we 
don't need to address all issues.

Problems
========

The first problem blocks usage, the rest mainly affect people heavily 
using shared collections and the dashboard view.  These latter problems 
aren't show-stoppers by themselves; the occasional confusing item 
popping to now doesn't make Chandler useless.  Unfortunately, the 
frequency of inaccurate pop-to-nows is high enough to make the NOW 
section's signal-to-noise ratio much lower than it could be.

1. Collection fails to sync [1]
-------------------------------

Esther's been seeing the main example of this [2].  What's happening in 
her particular case is the desktop and the server have different 
pictures of what an item's state is, so when the desktop sends a diff, 
the server rejects it for not containing enough information.  It'd be 
great to figure out why this is happening, unfortunately we haven't had 
much luck diagnosing the problem so far, but even if we can't completely 
stomp out this problem, we should fail more gracefully.

A fix to this could be to have the server's failure include a 
machine-readable error (right now there's text that contains the failing 
UUID, if we put that UUID in an invisible XML element, the desktop could 
try resetting that item and sending the whole item, not a small diff). 
There'd be a little bit of work on the server side, plus a fair amount 
of desktop sharing work.

[1] http://chandlerproject.org/Planning/OneDotZeroBugReset#CriticalBugs
[2] 
http://lists.osafoundation.org/pipermail/chandler-users/2008-January/001127.html

2. Mysterious pop-to-now [3]
----------------------------

There are a wide variety of reasons this is happening, it may be OK if 
we fix 90% of them rather than 99% of them.  With that in mind, my 
educated guess is that the lion's share of these could be fixed by 
adding a whitelist to the logic for deciding whether a sharing change 
should pop an item to now.  Right now, changes to most attributes (even 
if their values aren't visible to the user in the detail view) will 
trigger a pop-to-now.

Whitelisting attributes isn't 100%, occasionally it's ambiguous whether 
a change to an EIM field will cause a user-visible change, but there are 
many fields that clearly don't directly effect what a user sees in the 
detail view, these shouldn't be allowed to cause a pop-to-now.

A first cut at this would be fairly simple to implement on the desktop.

There are also issues with Chandler's understanding of email addresses 
being case-sensitive; it's not clear to me if these issues would be 
solved by white-listing, there may still be work for Brian Kirsch there.

[3] http://chandlerproject.org/Planning/OneDotZeroBugReset#PopToNowBugs

3. Inaccurate byline
--------------------

There are two main sources of inaccurate bylines I'm aware of: The first 
happens when two users have non-conflicting changes, but they sync 
out-of-order.  The byline is chosen based on the latest change, but the 
change time is based on when the user actually changed the item in 
Chandler, not when they synced.  So out-of-order syncs will cause the 
byline to stay the same, even though a new change has been merged in, 
which can be disconcerting for the person who synced first.

A solution to this would be to add additional fields to the ModifiedBy 
record, sending two users after a merge, and rendering an appropriate 
message in this case.  This is quite straightforward, but it might 
require a bit of server work.

The second main cause of in accurate byline happens when an item is 
deleted.  The server stores a "tombstone", recognizing that the item's 
been deleted, but it doesn't know *who* deleted it.  This isn't a huge 
problem for normal items (although it's a little annoying).  For 
recurring events, though, deletion actually means a triage status 
change.  In these cases, it's pretty confusing to see an occurrence pop 
to now with nothing but a triage status change and an inaccurate byline.

One solution to this would be to change morse code to include 
information about who's deleting an item, and having the server store 
that alongside the existing tombstone.  That might be a lot of work, though.

The recurrence problem could be fixed by changing the way triage-only 
modifications are sent to the server.  Unfortunately, changing this 
would either break compatibility with existing data, or require a huge 
amount of effort to maintain backward compatibility, so I think that's 
out.  This may just be a problem we live with.

4. Spurious conflicts
---------------------

Conflict resolution aims at preventing users from losing data. 
Unfortunately, there are so many false positives that I literally had 
never seen a *real* conflict until collaborating on writing this 
document!  All the other hundreds of conflicts I've seen have been 
spurious, adding noise to my Chandler use and forcing me to examine and 
click through too many conflict resolution dialogs.

Many spurious conflicts appear to come from having the same item shared 
in multiple collections.  When a user changes a multiply shared item 
locally and syncs, the first associated collection receives the item 
change.  The second collection then sends the same change back.  This 
ought to result in a no-op, but apparently it doesn't always (that's a 
bug in and of itself).

While ideally we'd fix the underlying bugs that are causing spurious 
conflicts, it's not clear to me how much effort it would take to solve 
the problem.

A radical, alternate approach that might reduce this pain, while still 
providing data preservation would be to

A) Store item state before every sync, and after every sync (we mostly 
do this now)
B) Provide an HTML picture of the history of an item, similar to the 
view currently available by pressing F4 but focused on reproducing only 
the attributes visible in the detail view
C) Auto-resolve *all* conflicts, with the more recent change winning, 
then use the banner button currently associated with conflicts to alert 
users if there was a conflict on a user-visible field

Implementing this strategy would be a moderate to a large amount of 
work, so it's not clear if it's worth doing, even if we agree it's 
desirable.  On the plus side, a clean-UI for the history of an item is 
an attractive feature in it's own right, so such an approach could be 
implemented incrementally.

Conclusion
==========

I prioritize these roughly in the order described.  Recovering from sync 
failures and whitelisting user-facing attributes seem relatively 
straightforward with a high value.  Expanding the ModifiedBy record to 
work better with merges seems lower priority but simple and valuable. 
Morse Code changes to deleted items, viewing item history and 
auto-resolving conflicts seem potentially valuable but possibly expensive.

Sincerely,
Jeffrey