[Chandler-dev] [Sum] June 12 - 18

[Katie Capps Parlante]

Build, release and QA
---------------------
*alpha2 patch*
I proposed that we patch alpha2 with a longer timeout to unblock 
dogfooders while we work on the root problems with cosmo-demo requests 
taking too long for large calendars. Philippe, Jared, Ted and Sheila 
agreed. Morgen had a patch handy.
http://lists.osafoundation.org/pipermail/chandler-dev/2006-June/006099.html

Sheila announced the new version of the milestone.
http://lists.osafoundation.org/pipermail/chandler-dev/2006-June/006142.html

*background sync*
The background sync branch landed in the trunk, so that more folks can 
test and fix problems. The trunk may be less stable as we work out the 
kinks. Stamping of previously shared items is disabled. Shared 
collections will sync every hour; you can change the frequency using a 
preference. The "Sync All" toolbar button initiates a sync. If you find 
bugs, please document the steps to reproduce the bug and include 
chandler.log + twisted.log.
http://lists.osafoundation.org/pipermail/chandler-dev/2006-June/006112.html

Based on Morgen and Andi's recommendation, I proposed that we leave 
stamping disabled for shared items for alpha3, to focus on stability. 
Sheila agreed with the proposal.
http://lists.osafoundation.org/pipermail/chandler-dev/2006-June/006135.html

*alpha3 code freeze approaching*
Alpha3 code freeze is planned for Wednesday, June 21. We'll restrict 
checkins to regression fixes, considering other fixes on a case by case 
basis. We'd like alpha3 to be more stable than alpha2. We have two big 
changes landing (background sync and a new wx tarball) that are likely 
to cause regressions.
http://lists.osafoundation.org/pipermail/chandler-dev/2006-June/006114.html

*new binaries for linux*
Bear enabled the new linux binaries, which are now being built on Ubuntu 
(Breezy Badger). You may run into problems if you are on FC2. Two 
problems we've seen so far: our wx links to libsdl and libexpat and some 
distros don't have them. To prevent problems, remove chandler/release 
and chandler/debug directories before doing a make install.
http://lists.osafoundation.org/pipermail/chandler-dev/2006-June/006118.html

*Ubuntu 6.06*
Andi got the ubuntu build to work after installing more packages. 
Chandler crashed easily with a gcc/gcj build, but was stable with 
gcc/gcj 3.4.6. Chandler looks ugly because the font sizes are off. Andi 
found the ubuntu install to be the easiest he's experienced yet.
http://lists.osafoundation.org/pipermail/chandler-dev/2006-June/006125.html

*Bug 6040 blocking progress*
Sheila raised the priority of Bug 6040 to P1 and asked that we not sit 
on this bug, as it prevents the downloadable version of Chandler from 
running. Grant explained that the share directory doesn't show up in 
chandler/release on a full build of the app, due to an i18n bug. Markku 
explained that Jeffrey checked in a workaround, and that Brian Kirsch 
would tackle the root problem when he's back from vacation.
http://lists.osafoundation.org/pipermail/chandler-dev/2006-June/006133.html

*Tracking performance*
Heikki explained how we continuously track Chandler's performance. He 
gave a link to 30 day trend graphs, the Tinderbox status page 
(displaying the latest results from automatically run performance 
tests), the performance project home page (describing the 6 out of 18 
tests that we are prioritizing), etc.
http://lists.osafoundation.org/pipermail/chandler-dev/2006-June/006130.html

I responded to an earlier post from Heikki about performance tests, 
noting that we have conflicting goals: run tests reasonably quickly, run 
tests against historical data sets, run tests against data sets that 
more accurately reflect user data, run tests that cover the full app 
including expanding functionality. Heikki replied that he is happy with 
the current times.
http://lists.osafoundation.org/pipermail/chandler-dev/2006-June/006140.html

*Logging discrete actions in tests*
Dan asked Heikki about his removal of logger calls from sync tests. Dan 
assumes we want to be reporting about discrete actions from within a 
given test. Heikki explained that he found the logger start/stop calls 
confusing. He thinks the goal is not to report discrete actions, but to 
report errors as accurately as possible and with good enough information 
to find problems. Heikki explained this particular test in more detail, 
that the exceptions give the relevant detail.
http://lists.osafoundation.org/pipermail/chandler-dev/2006-June/006137.html

*external tools*
Markku brought up a discussion about how to distribute tools with 
Chandler (gettext, wx tools, etc.). Two options: (1) New build target 
downloads prebuilt binaries of tools. (2) We obtain sources from all 
required external tools and place them in external. (1) is fast, but 
more burdensome as # of platforms we support increases. Note that on 
Linux, the binaries are not always portable. (2) adds bloat to the 
project, but tools will work across all platforms. Any other ideas?
http://lists.osafoundation.org/pipermail/chandler-dev/2006-June/006139.html

John proposed #1 for Mac/Windows and #2 for Linux. Reid modified the 
proposal: #1 for Mac/Windows/Ubuntu and #2 for other Linux. Heikki 
pointed out that the real need here is i18n tools, and proceeded to 
explain the tools in more detail.
http://lists.osafoundation.org/pipermail/chandler-dev/2006-June/006145.html

Intern and SoC projects
-----------------------
*Natural language parsing in Chandler*
Philippe liked the NLP plan put forward by Jeffrey and Darshana. 
Philippe and Grant noted earlier discussions and designs where the 
search field is re-purposed for a rudimentary CLI, including Mimi's 
QuickItemEntry.
http://lists.osafoundation.org/pipermail/chandler-dev/2006-June/006093.html

Heikki brought up Firefox's search box as an example. He noted that 
being able to change the context (from "search" to "new event") might be 
useful, particularly with a small set of contexts. Jeffrey pictures a 
drop down, where all commands are discoverable. The drop down could have 
icons as well as the command. By switching contexts, the end user would 
be able to create a bunch of tasks just by entering them and hitting 
return (as requested on the design list). Search mode would likely be 
the default. Jeffrey painted a picture of being able to enter tasks with 
dates quickly from the keyboard (no mousing around). Bear brought up 
QuickSilver as a nice example handling contexts from the keyboard.
http://lists.osafoundation.org/pipermail/chandler-dev/2006-June/006095.html

Davor finds the Firefox search box awkward, as it requires using the 
mouse. He's in favor of all-keyboard entry and likes the "switch 
context" idea.
http://lists.osafoundation.org/pipermail/chandler-dev/2006-June/006131.html

Ted gave another use case: copy and paste text from an email and have it 
be recognized. Ted is not a fan of the CLI approach (at least not long 
term). Agenda and Apple didn't need a semantic hint. He noted that Todd 
Agulnick wrote some of the recognition code in Agenda. The Newton 
allowed you to select text and tell it to "recognize" it -- it would 
create the appropriately typed item.
http://lists.osafoundation.org/pipermail/chandler-dev/2006-June/006104.html

Heikki threw out the idea of "recognizing" text as it was typed, 
underlining items that it recognized. It could automatically add in 
fields, or possibly automatically stamp the item. He also noted that you 
probably want to start out with a new "note" and have it adjust the 
type/stamp as it recognizes, not based on the app mode in the toolbar.
http://lists.osafoundation.org/pipermail/chandler-dev/2006-June/006103.html

Ted suggested that we not feel bound by NLP toolkits. We could consider 
how to use the domain model to help disambiguate parts of the 
recognition steps. Ted gave feedback that the parsing resource method 
could take hints to prime the recognizer to find particular types/fields.
http://lists.osafoundation.org/pipermail/chandler-dev/2006-June/006101.html

Philippe pointed out that Ted's Newton and Agenda examples didn't have 
to distinguish between "search" and "create new item" -- there is no 
particularly good way to do this (example: "tomorrow morning"). He 
agreed that it makes sense to have a more generic "create" context 
instead of having one command per kind. Philippe liked the proposal that 
was developing, to display the current intent and allow it to be changed 
through keyboard commands.
http://lists.osafoundation.org/pipermail/chandler-dev/2006-June/006116.html

*Multivariate Analysis*
Philippe and Markku had a dialog with Xun about his project, discussing 
the appropriateness of slashdot feeds as the data set, adding semantics 
to the data (to be more similar to PIM data, which does have semantics), 
statistical methods used, and use of Lucene. Andi recommended 
java-user at apache.lucene.org for Lucene usage advice, as well as the book 
"Lucene in action". Xun will check his work into the sandbox.
http://lists.osafoundation.org/pipermail/chandler-dev/2006-June/006110.html

Alpha3 and Alpha4 development
-----------------------------
*Stamping and Dashboard domain-model questions*
Bryan had some questions about stamping and the dashboard. (1) Are we 
using annotations for stamping? How do I know what stamps an item has? 
How will the attribute editor mechanism need to change? (2) What happens 
to attribute redirection? How do we implement "Who" and "Date" columns? 
(3) Does column sorting always imply an index on each user visible 
collection? Given the factors that affect sorting, we could have a lot 
of potential index updates. (4) The triage column sorting requirements 
imply a triageValueChangedDate attribute -- this may imply a similar 
repository level mechanism to set and update date-created and 
date-last-modified attributes. (5) The "purge" feature implies a 
separate attribute to store the pending triage state. Are there issues 
with conflicts here? (6) How do we implement the feature where the Send 
button is enabled when the user edits a received email event? (7) How 
are user visible To/From/CC fields managed for items that could be 
coming and going?
http://lists.osafoundation.org/pipermail/chandler-dev/2006-June/006097.html

*Filters loading items*
Andi explained a long standing performance bug where some filter 
expressions used by FilteredCollections use an expression that cause an 
entire item to be loaded into memory. This becomes a problem in some 
scenarios that iterate over the collection -- all items in the large 
collection get loaded. Two collections have this property, related to 
reminders and recurrence. Their filters require some other items to come 
up with a True or False answer. Andi proposed a 'bequeatheTo' feature, a 
reverse of 'inheritFrom'. Whenever a value in a particular attribute 
changes, the attributes in its 'bequeathTo' aspect would be given the 
value. This would enable values to be cached on the item tested by 
FilteredCollection. These attributes can be looked at without loading 
the item.
http://lists.osafoundation.org/pipermail/chandler-dev/2006-June/006105.html

Grant and others looked into this, and decided bequeathTo isn't needed 
in this case. Grant gave the code snippets for the two cases, and 
explained that one test can be dealt with using the existing 
findValue(). The other two tests are really trying to figure out if a 
refcollection is empty or not, and Andi will add API to the repository 
layer for that purpose.
http://lists.osafoundation.org/pipermail/chandler-dev/2006-June/006106.html

Jeffrey gave more details about bequeatheTo. The idea came out of a 
conversation between Andi and Jeffrey, the intent was to use it as a 
performance work around for issues with indexing new union collections. 
Andi and Jeffrey have been discussing a sub-index feature that should 
help avoid re-indexing. That improvement may be enough to make the 
bequeathTo concept unnecessary.
http://lists.osafoundation.org/pipermail/chandler-dev/2006-June/006108.html

*Performance improvements*
Andi reported that the filter problem noted above has been fixed: no 
filter expressions are currently causing items to be unnecessarily 
loaded. Andi also found and fixed another problem causing items to be 
loaded when iterating item uuids of a collection. These fixes yielded a 
performance boost for view switching and calendar overlaying (these 
operations create adhoc indexes). Importing a 3,000 event calendar on an 
intel mac sped up by 79%.
http://lists.osafoundation.org/pipermail/chandler-dev/2006-June/006121.html

Jeffrey reported on further performance improvements resulting from his 
work with Andi. He's using Freeze()/Thaw() on wx column headers when 
changing their text, which makes the Windows/Mac rendering faster. As we 
change the headers for each day of the week, we were repainting many 
things for each label change. He was able to optimize event lookup for 
"this week" by assuming most events are not longer than a week, handling 
this as a special case. He's using Andi's new sub-index support to 
optimize finding recurring events for "this week". These fixes help the 
case of "jump to next week". Jeffrey gives some perf #s in the writeup. 
He noticed that the speedups are different on Intel Mac and his Dell; 
this could be explained by Mac's faster harddrive, if iterindexkeys is 
IO bound. Jeffrey also noted different performance characteristics for 
recurring events and non-recurring events, suggesting that we match our 
test data with "normal users".
http://lists.osafoundation.org/pipermail/chandler-dev/2006-June/006126.html

*Recurrence in sharing*
Jeffrey has been thinking about recurrence and sharing, while looking at 
rewriting some import code. To preserve data integrity, Jeffrey thinks 
we should move recurrence conflict handling out of the ICalendarFormat 
import/export layer and deal with it at a higher level in the sharing 
process. Morgen has been understandably reluctant to add that complexity 
to sharing. Jeffrey thinks the recurrence bugs will become more 
confusing and difficult if not handled more directly in the sharing 
layer. Jeffrey mentioned that ideally we'd have a journal of server 
changes to a recurring event and then compare them to a journal of 
changes since the last sync.
http://lists.osafoundation.org/pipermail/chandler-dev/2006-June/006109.html

Morgen was open to doing whatever we need to do to make this work. He 
asked for a summary of current recurrence sharing problems. He asked 
about his two recurrence sharing bugs, and if we should fix them using 
the current approach or with a new mechanism.
http://lists.osafoundation.org/pipermail/chandler-dev/2006-June/006127.html

Meetings and announcements
--------------------------
*Apps meeting*
http://wiki.osafoundation.org/bin/view/Journal/AppsMeeting20060615