[Design] recommendations from sister project...

[David Jeske]

I have a personal project called Wintermute which is a sister-project
to Chandler. In the interest of helping the folks working on Chandler,
I'd like to share some lessons that I've learned so far.

Background:

When the first OSAF press went out, I was inspired to install Lotus
Agenda, and what I saw was really neat. I had already been working on
a tool in this space in the past and seeing Agenda inspired some new
ideas. This was last August, between the Vista prototype and the first
Chandler development release.

While I have lots of experience with Python, I like using strong typed
interfaces when software needs to be developed by many parties
(i.e. when it has plug-in apis). I decided to use Java Swing for my
app. I also have a strange death-wish in wanting to know if Java will
ever really work for client software.

You can flip through some screenshots here, although one of the nicest
things about Wintermute is the threaded-interactivity and UI
responsiveness, and that dosn't show on the screenshots.
 
  http://www.neomason.com/wm.cst

Architecture Details:

 - Wintermute uses an object database storage model with propeties
   and relationships between objects, the most interesting feature
   is that all relationships are bi-directional. This means that
   it's always possible to track backward links, and it's easy to
   break links from either side of the connection. This is possible
   because the relationships use a reciprocal naming scheme. If
   adding the "email:contact" relationship on one side, a relation for
   "contact:email" is added on the other side.
 - It uses sleepycat transactional server for the raw storage.
 - With lots of relationships created between items, and all of
   those relationships requiring both backward and forward links,
   the potential to do lots of writing exists. Wintermute minimizes
   the amount of time spent doing this by using a checkpointing
   scheme. All changes occur in memory only. Periodically, one
   transaction is opened and 
 - The user-inteface is defined at a high-level by the objects
   themselves. It's similar in concept to XUL, but only exists in the
   object database, not any flat XML file. This is neat, because
   storing configuration and state data about UI instances is really
   easy. For example, all tables remember exactly where they were
   scrolled, what was selected, and allow the same level of
   configuration.
 - It has some nice general models for synchronizing with external
   datasources. Currently implemented are: UNIX mbox, POP, IMAP, CSV,
   MySQL, RSS, and Yahoo Intellisync (has some bugs). 
 - It has generic table, email, and calendar data viewers
 - It uses lucene to full-text index and search over all data
 - A background "relationship builder" watches all object changes and
   periodically builds relationships, handles full-text indexing etc.

Lessons:

 - Life will be better for everyone if there are separate 'databases'
   for storing (a) user-authored data, (b) remote synchronized data,
   and (c) application configuration.

   Because Wintermute is in flux, we are constantly dropping the
   database and rebuilding to pick up new application UI configuration
   data. This causes a problem because it drops all the user data and
   remote synchronized data at the same time. This issue will also
   exist when new application releases are distributed which have new
   UI metadata options. We really need to support separate
   repositories for these different types of data.

   It is also obvious that in the future it will be important to
   provide a simple backup for the user's originally authored data.

 - bi-directional relationships are good. Wintermute relationships are
   a little too sloppy because their names are totally
   arbitrary. Building names out of the fields which cause the
   relationships is probably more sensible. This would allow the 
   background relationship builder to easily do it's work by merely
   being provided the object and fields which changed. Currently
   the wintermute relationship builder can add new relationships but
   it dosn't have enough information to break old relationships.

 - building everything on the same object storage machinery has some
   really nice consequences. The Wintermute code which maps objects
   and object relations to Swing TableModel and TreeModel interfaces
   was only implemented once, however it is used underneath
   everything. Changing object data anywhere in the UI will instantly
   update any visible occurances of information based on that related
   data.

 - Even though synchronization is important. Too many compromises are
   made when presenting data in it's "natural" foreign format. For
   example, synchronizing with an IMAP server and displaying all the
   IMAP folders is easy, but it forces the UI in a direction of being
   just like old stuff.

 - It's good if objects know their types. I've read interesting
   discussion about whether or not objects should have only one type,
   or have many types based on how they are used. I'm wavering back
   and forth on this. So far wintermute objects have only one type,
   and if you want an email to turn into a repeating calendar event,
   you create a separate repeating calendar event object and just link
   it to the email. However, it seems easier if objects know their own
   types.

 - a strong distinction between raw "external-data" and the places
   where that data is integrated locally simplifies many
   things. Synchronizing with external datasources in wintermute is
   really easy, because we represent all the external data inside
   wintermute in its natural format. In a way, we've divided the
   synchronization process into two steps (a) replication, and (b)
   correlation.

 - checkpointing and transactions make a very powerful combination for
   saving lots of data changes quickly. Wintermute makes tons of changes to
   little objects all the time, yet even when it's synchronizing, the
   checkpoint process is relatively quick. 

 - supporting two "fancy clients" which are synchronized when the
   server is not "fancy" is really hard. (i.e. two Wintermute clients
   with an IMAP server for email) 

 - Java Swing's APIs are nice, but their layout model is wrong for 
   apps like Wintermute. Everything got easier when I started writing
   my own simple layout managers.

 - No matter what anyone tells you, sweeping GC causes random pauses
   which are annoying. Python's reference counting should be better,
   because deref collection pauses will occur during an action in a
   predictable manner.

 - Java Swing is still slow, and Java is a CPU/memory hog.

Hope some of that helps.

-- 
David Jeske (N9LCA) + http://www.chat.net/~jeske/ + jeske@chat.net