[Dev] Weekly Status Report 10 October 2003

[John Anderson]

*Last Week:*

    * Plan: Finish removing ZODB from Chandler, except where features of
      the repository makes it impractical.
    * Actual: Done. See detailed repository discussion below
    * Plan: Follow-up with Andi to resolve remaining repository problems.
    * Actual: Done. Andi, Brian, Michael, Katie, Stuart and I all met
      for a very productive meeting in which we came up with good
      solutions for all the technical problems.
    * Plan: Follow-up with Katie on the design for parcel loading.
    * Actual: Done
    * Plan: Follow-up with Jed on his progress with document editing.
    * Actual: Done
    * Plan: Attend another data meeting.
    * Actual: Done
    * Plan: Work on the design for programming CPIA blocks.
    * Actual: Made a small amount of progress thinking through a design
      proposal.

    * Also: Met briefly with Ducky to discuss documentation for the
      repository.

    * Repository Discussion: I'm now able to run Chandler and see the
      MrMenus viewer parcel build completely on top of our repository.
      I've also converted the calendar viewer parcel (however it's not
      fully debugged), but none of the other viewer parcels. In the
      process of using our new repository I ran into the following snags:
    * I spent a few days tracking down a particularly thorny repository
      bug. The symptom was that occasional random items were not being
      persisted, but it wasn't always reproducible. It turned out to be
      caused by ZaoBao using the repository in a different thread. The
      repository turned out not to be thread safe. I suspect similar
      bugs to occur with other parts of the code that use threads, e.g.
      Notifications. Until the repository can be used from multiple
      threads, we proposed throwing an exception when use occurs from
      different threads.
    * When using Items without pre-existing schema, it isn't possible to
      add an item reference to a list. Doing so causes the item
      reference not to be persisted. The situation comes up in numerous
      places in Chandler. Currently it has the consequence that running
      Chandler without an empty repository fails. We have a proposed
      solution which built on ideas that came up in last week's
      repository meeting.
    * Lists in our repository are currently implemented as bidirectional
      linked lists. One consequence of this design is that large lists
      can be sparsely loaded. Another consequence is that lists cannot
      be randomly accessed by indexing as in Python lists or arrays in
      other languages, e.g. to get to the n'th item you must follow n
      references. This design precludes efficient use of some common
      algorithms that depend upon indexing, for example binary search
      and quick sort (e.g. the Python list sort function). Because
      binary search comes up so often in sorted collections, I believe
      we will end up needing to implement a list that can be indexed
      quickly. There are no plans to implement this kind of list.
    * One of the advantages of Python, and other interpreted languages,
      is the ability to store references in collections and lists -- in
      fact there are no restrictions on where you can store different
      types of data. Our repository design differs from Python, in that
      it only allows for references to be stored in special reference
      collections. Eventually I think this will end up being too
      restrictive.
    * I implemented a simple solution to the schema evolution problem
      for Viewer's model data using ZODB's __setstate__ function. This
      solution depended upon the ability to migrate the data to a new
      schema from within __setstate__. Our new repository doesn't allow
      this, so I need to come up with a different solution.

*Next Week:*

    * Plan: Incorporate changes to the repository that allow Chandler to
      run on the new repository with MrMenus and Calendar. I don't plan
      to convert the remaining Viewer Parcels to the new repository
      because Viewer Parcels are going to be superseded by CPIA
      documents. However, I will send out instructions in e-mail on how
      to do the conversion for anyone who is interested in converting
      their Viewer Parcels.
    * Plan: Check-in my changes to Chandler to use the new repository.
    * Plan: Solve the schema evolution problem as well as it was solved
      using ZODB in the previous design.
    * Plan: Remove the mountains of old repository code in CVS.
    * Plan: Attend another data meeting and discuss the repository
      issues described above.
    * Plan: Work on the design for programming CPIA blocks.
    * Plan: Follow-up with Katie on the design for parcel loading.
    * Plan: Follow-up with Jed on his progress with document editing.