[Chandler-dev] [Sum] June 26 - July 2

Katie Capps Parlante capps at osafoundation.org
Mon Jul 10 13:51:47 PDT 2006

Build, release and QA
*wxPython build*
Bear sent out wiki notes about the wxPython build, after recent changes.
The notes explain how to build wxPython from external/wx or from a
working directory of the OSAF wx repository. Comments and questions
welcome (wiki or list).

*Weekly checkpoint*
Bear sent the checkpoint for 26 June:

Dan sent the checkpoint report:

*Unicode error on Ubuntu Breezy*
Bear pointed out an Account dialog error on the trunk that shows up in
functional tests ("Failed to convert file contents to Unicode"). You can
press "Ok" to continue, or use a workaround from Robin (until it gets

*Alpha3 schedule slip*
Given a bug count of 19 bugs requiring code changes on Monday Jun 26 
(also with holiday  and vacation schedules), we're moving the Alpha3 
final date to Tuesday, July 18. Code review before commit is now 
required. We'll branch when we hit 10 bugs requiring code changes (this 
doesn't count test tasks, doc tasks, etc.) We were at 11 such bugs on 
Friday Jun 30, hoping to branch the next week.

*Intermittent certstore failure*
Heikki explained the certstore functional test fails intermittently
(once every few days). It assumes the order of the certs in the summary
view is always the same. Speak up if you have information about why the
order would change. Heikki is tracking this with Bug 6157.

*Tinderbox status app*
Robin wrote a small wx app that displays Tinderbox status (cross 
platform of course). Bear suggested 300 second updates instead of 30 
seconds -- even the fast builds take more than 20 minutes.

*Alpha3/Alpha4 priorities*
I listed priorities during the end-game for alpha3. (1) Fix bugs that 
prevent alpha3 from being a dogfoodable calendar with working background 
sync (2) Functional and unit tests (3) Alpha4 work: experimental 
dashboard and scheduling, performance, data migration, etc.

Sheila sent links to all of the specs that apply to alpha4, many of them 
recently updated:

*Adding check() to functional test suite*
Heikki observed that a corrupt repository had gone undetected by a 
functional test run (though the script was unable to close the 
repository). Bryan experimented with adding check() after each test in 
the run (adds 2 minutes to the test run) and with adding check() only at 
the end of the run (adds 2.5s). Bryan is sensitive to the time the test 
runs take, and asked if there is a big benefit to running check() for 
every test.

Andi noted that in DEBUG mode, values are checked during assignment and 
check() is run upon quitting. Running check() after every test has the 
benefit of detecting automatically which test caused the repository 
corruption. Given that check() is expensive, Andi suggested adding it at 
the end of the test suite run.

Intern and SoC projects
*Collecting Usage Data/Central Logging*
Ashkan is an intern working with the PPD team. His project is to
instrument usage tracking in Chandler to understand how dogfooders use
the app. He's been working with John Anderson to log CPIA events. Users
can enable/disable the logging via the test menu (and are given some
privacy disclaimer). The logs either go to the filesystem or some remote
logserver. Ashkan asked for the best (easy) way to collect the data --
he'd like to focus on the analysis. He's considered HTTP, rsync/ftp,
Chandler's background sync, and SMTP.

Bear noted that Ashkan is right to be concerned about outgoing ports and
transport mechanisms -- without a server listening to a known port, most
users won't be able to use the service. Some email servers limit
outbound email traffic and would require a configuration step to
authenticate to the SMTP server. Bear also noted that it might be wise
to encrypt the stream or the data given the personal nature of PIM
information. Bear also cautioned about the load on the server. Bear
doesn't think rsync sounds practical as all of the data is being
generated new on the client side -- HTTP Put or WebDAV may be the way to
go. Bear thinks sftp/ftp are non-starters -- too many security issues.
He suggested XMPP and a pub/sub setup as an alternative.

*Contacts/Address Book*
Grant suggested carving out time to come up with a plan for fields with
multiple values (home/work/etc. phone#) -- there is work here at the
domain model layer, the detail view, etc. Grant also suggested working
on vcard import before stamping. A contact stamp might not be that
useful right away; importing existing contact data with the right fields
will be useful early on.

Ted agreed with Grant's suggestions, and brought up the issue of linking
contacts with other items (tasks assigned to people, people attending
events, etc.). Ted brought up the question of where contacts appear in
the UI, questioning the usefulness of having contacts show up in the
"All" collection. He suggested an "address book" collection as a more
useful short term solution. He encouraged a discussion with Mimi about
the design.

Bear also recommended starting with vCard import first. He suggested
that contact groups be handled using Chandler's existing Collection
mechanism. He seconded Grant's observation that handling multiple
occurrences will be important. Bear noted that he's excited for contacts
to show up as this is one of his personal dogfood requirements.

Bear sent a link to Dan Connolly's articles about parsing and extracting 
information from iCalendar, vCard, hCard and other semweb/microformat 

*Natural Language Date Parsing*
Darshana explained to John that dateutil doesn't parse "natural
language" -- the API from Bear's project is closer to the one that we
want. Darshana proposed wrapping dateutil with an API that looked more
like Bear's API.

Bear asked if Darshana knew of other bugs/problems beyond one they had
discussed earlier. He noted that he could alter the API of his library
so that it returns the needed datetime types. Converting should be easy.
Bear noted that Darshana is the first user of his parser, so she has his
full attention and help. Patches are welcome.

Dev Projects
*Installers and data migration*
Sheila noted that the data migration and installer plans are a bit up in
the air, given changes to the 0.7 release schedule and our dogfood
experience using the sharing server as part of a manual migration path.
The 0.7 release has expanded into the beta release, adding more alpha
releases. In alpha4 or so, we'd like to get users starting to play with
the dashboard and tasks, so it will be important to migrate more than
calendar data. Our small number of dogfooders are going to move along
with us on the alpha releases, so we don't expect to have to move people
from 0.6.1 --> 0.7. Given the role the sharing server plays in the
migration path, we'd also like to invest in making it easier to go
through those steps with multiple collections. Sheila outlined some
requirements for users installing a new Chandler, migrating their data
and getting set up against a new server. She asked if there are
difficulties in backing up other kinds of data (account settings, prefs,
tasks and email, stamped items, etc.) She asked if it was too ambitious
to do this automatically.

*Recurrence in sharing*
In response to Morgen's query about problems we face with sharing
recurrence, Grant explained Jeffrey's idea. For recurring events (either
locally or on the server) don't make changes to the local object
directly but have ICalendarFormat handle the changes and the merge (by
saving changes and replaying them at the appropriate time). The
justification is that recurrence is complex and sort of nasty -- it
creates a graph of interdependent objects. It is difficult/error-prone
to merge the two sets of changes to a general object graph without
recurrence domain model knowledge. The ICalendarFormat approach is
tricky enough to not be recommended for alpha3 (unless it solves
yet-to-be-found intractable recurrence-related sharing problems).

Andi asked if the current recurrence design and implementation is too 
complicated. Should it be revisited and simplified? He noted that the 
repository needs to be correct in its merging, regardless of the 
complexity of the graph changes. From the repository point of view, the 
complexity of recurrence is a great test case to flush out weird edge 
cases. He acknowledged that the merge could be wrong from a semantics 
point of view, having no knowledge of the app semantics -- in this case 
Jeffrey's proposal may be required.

*Stamping and dashboard domain model*
Grant answered some of Bryan's questions. Annotations for stamping?
Grant wrote some exploratory code, but has put this on hold to work on
recurrence for alpha3. How to find out what stamps an item has? Grant
explained you'll be able to ask an item for its stamps, PJE gave example
code. How to specify an attribute? Perhaps a dotted attribute name,
although perhaps some alternative if we decide this exposes
implementation details. Attribute redirection? Duck-typing? We're moving
away from this, instead fixing which attributes are displayed in which
view. If we move to using onValueChanged() and we implement stamping
with annotations, then some onValueChanged() implementation needs to
iterate over stamp classes. Repository index on each user visible
collection? Too many index updates? Grant explained that sorted tables
require an index. Perhaps we need the ability to index and update
computed attributes (a repro version of Python properties). Perhaps we
can leverage subindexes. Mechanism for updating an attribute when
another attribute changes? Only onValueChanged(). Monitors seem like
they'd be overkill. Purging indeed looks tricky wrt auto-reindex-columns
paradigm. How to detect "sendability" in the UI? The info should live in
the domain model, but perhaps not quite right to track sendability for
all items. Could the UI attach a monitor for communication items? User
visible to/from/cc relationship to RFC822? Grant will go think about
this after reading spec again more carefully.

Bryan made a list of domain model requirements for the dashboard work.
He also has a wiki list of tasks and open issues. A few highlights: How
to handle indexes to support sorting, given calculated fields that don't
just reflect a simple attribute? How to model ticklers? Triage status?
What is the architectural connection between columns and sectioning? See
the full message for the complete list.

Andi suggested using an AttributeIndex feature where multiple attributes
can be used to sort. Dynamically changing the second attribute's sorting
could be expensive. Bryan explained that the ordering requirements are 
complex: Unread/Read first sort, then Created/Edited, then First/Update, 
then triage status, etc. The second one includes Kind-dependent 
attributes. Bryan can write a function or set of functions, hence the 
questions about 'calculated' sort attributes. Andi asked about 
'Calculated' attributes. Bryan and Grant gave CalendarEventMixin's 
"endTime" as an example (calculated from startTime and duration). 
Calculated is a subclass of Python's builtin property.

Bryan and Grant started a side discussion about fields used by 
Calculated -- schema type and a list of dependent attributes. The schema 
type is used by attribute editors to pick what editor to use. The 
dependent attributes are used to set up notifications to keep the UI up 
to date. We could make use of this to update indexes when dependent 
attributes change. Blocks can set up notifications by defining a 
viewAttribute (the attribute to watch) and an onItemNotification method. 
If the Block has an item in its "contents", then onItemNotification will 
be called if the attribute named in viewAttribute changes. If more 
complex behavior is required (more attributes, or attributes on other 
items) the block can override Block.getWatchList. Bryan noted that there 
are examples of overrides in Detail.py -- "all day", updating the 
start/end time as you drag an event around the calendar, etc. He gave 
more details bout the "all day" attribute -- 7 blocks that need to 
update themselves when this changes. Bryan explained that one should be 
careful not to change attribute values in the onItemNotification method.

Andi noted that the collection order can be modeled by a ref collection 
or a numeric index, in which case the order can be persisted and 
arbitrary (reflecting the user's choice). Bryan explained that his 
concern centers around adding manual ad-hoc additions to an existing 
sort order. Andi suggested that the last criteria by which items are 
sorted could be the manual user choice. He didn't see a good way to 
allow arbitrary user order injections anywhere in the index. Andi asked 
if it would be helpful to think of it as displaying two collections: 
first the user sorted one and then the program sorted one.

*Repository documentation*
Markku asked Andi for a good overview of the repository implementation. 
Andi pointed him at his blog, and noted that there are no comprehensive 
design docs. Andi asked for ideas about specific blog posts he could do. 
Markku noted that the split between the python and c code makes it 
harder to read and understand the code.

Heikki asked for a short introductory document on the repository, 
written at a high enough level that it doesn't become obsolete. He 
suggested including high level requirements and goals (including 
performance targets), high level descriptions of each of the modules, 
and explanations of the core concepts (kinds, items, attributes, 
monitors, watchers, views, etc.) Andi said that if someone started such 
a document, he'd review, edit and enhance it.

Brian Kirsch is taking the next leg of the i18n work: streamline 
internal egg loading code, external API, integrate egg layer with OSAF 
code, figure out how to package and distribute localization eggs. We 
should be at a point where we can start localizing in a couple of weeks. 
  Other required pieces that should be showing up in that timeframe: 
callback to XRC dialogs to use Chandler MessageFactories, wxPython 
distribution changes to handle localizations without Chandler i18n code.

Meetings and Announcements
App team meeting:

We did a second IRC test session Friday, Jun 30 focusing on bg sync:

More information about the chandler-dev mailing list