[Chandler-dev] [Sum] The Great Architecture Discussion of 2007

[Philippe Bossut]

Hi,

Thanks Grant for this summary. I'm going to comment on each of the main 
point. I assumed that, in the absence of an *Agreement* section, the 
subject was still open though it doesn't mean we are collectively in 
strong disagreement.

Grant Baillie wrote:
> Rearchitecture vs End-User Feature Development
> ----------------------------------------------
>
> There was some concern (e.g. from Brian K, John, Andi) that a major 
> rearchitecture/rewrite will time away from developing end-user 
> features, and fixing bugs that impact end-users (i.e. being responsive 
> to our users). This would in turn hinder adoption, which could have 
> disastrous consequences for the project.
>
> *Agreement*: Having everyone "drop tools" to work on rearchitecture 
> (i.e. longer-term goals) would not be good.
> *Agreement*: There was some support (Jeffrey, Morgen, PJE, Heikki, 
> John) for having a smaller group attack architecture on the side, 
> possibly starting out with CPIA. PJE pointed out that this could be 
> done similarly to previous work of his at OSAF (schema API and EIM), 
> where a couple of developers were involved initially, before others 
> moved over to the new architecture as needed.
> *Agreement*: (PJE, Paul) Focusing only on getting end-users will not 
> be enough to make the project sustainable once funding expires.

Add me to the list of people who are in agreement with those *Agreements*.

On the idea of having a "small group attack[ing] architecture on the 
side", my concern is to make sure that the progress of this work is 
correctly tracked and focus on the right thing. My experience with this 
kind of project (2 in my past life as a manager and both unsuccessful) 
makes me prudent and wary: it's easy for the "rearchitecture guys" and 
the "product maintenance guys" to be so disconnected that the 2 projects 
suffer and fail, the "rearchitecture project" going on a wild goose 
chase with grandiose objectives, the "product maintenance" seeing its 
immediate needs (say testability or performance) not addressed in any 
foreseeable future and growing downright negative on the rearchitecture. 
This is a very serious risk.

To mitigate that risk we'll need to get a monthly formal status point, 
reviewing what has been learned in that month, what's the next action 
and how more clarity the recent progress gives us on the overall 
schedule of the project. The rearchitecture project will have to be very 
open so that the rest of the team stay engaged and on board with the new 
architecture. We also have to have enough visibility that, if the goals 
seem to slip further away every months, we should be able to call the 
project off and cut our losses.  This is what could be called an 
"accountability" clause for the rearchitecture project.

> Testability
> -----------
> PJE: Testability is a requirement for the goals laid out in Katie's 
> original email, and is therefore a "wedge issue" for the whole 
> discussion. Testability is also necessary to build a developer community.
> Morgen: Testability has been vital for development and ongoing 
> maintenance and feature development of syncing code.
> Heikki: Testability has not been a requirement in the development of a 
> well-known family of open source web browers whose process he happens 
> to be familiar with.
> There was some discussion of testability in the thread following 
> Aparna's "Desktop Test Automation Project" email. In the standard 
> layered approach, we could tackle testability of presentation layer 
> code with a mock wx.

I can't pull myself and say that testability in and on itself should be 
the driver. IOW, if testability was our only issue with the 
architecture, we couldn't justify overhauling the architecture just for 
that and would rather think about other approaches to test (test by 
community, use off the shelf testing frameworks, etc...).

> Architecture and goals
> ----------------------
> PJE's desirable architecture: (i.e. separated layers, for testability 
> and pluggability) is the old spike project he did a couple of years 
> back. (Some of these ideas have since been incorporated into Chandler):
>
> http://svn.osafoundation.org/chandler/trunk/internal/Spike/src/spike/overview.txt 
>
>
> PJE's 3 architecture process options, quoted:
>
>> 1. Start with the current architecture and evolve it in-place
>> 2. Define a new architecture in terms of "off-the-shelf" Python 
>> components
>> 3. Develop an architecture specifically to suit Chandler's current goals
>
> The first minimizes upfront costs (rearchitecture) vs less apparent 
> later costs (time to stabilize releases).
>
> Andi, John, Heikki: Favour incremental vs more radical architecture 
> changes.

Add me to that list of 3 folks. Does that make "Favour incremental 
changes" an "Agreement" point? See also my point on "accountability" 
here above.

> CPIA/Persistence in the UI
> --------------------------
>
> Katie's original email suggested that we shouldn't be persisting so 
> much of the UI structure of the app. John noted that transparent 
> persistence was a refreshing contrast to other systems he'd worked on, 
> where you had to write SQL queries every time you needed to persist 
> something. Philippe pointed out that having everything be persistent 
> is confusing for new developers: it's hard to make sense of all the 
> attributes that get tied to even simple items when displayed in the UI.
>
> PJE: What we need is to separate out visual presentation (not 
> persistent) from application logic (e.g. which items are selected in 
> which collections, etc). That leads to greater testability (you can 
> test the application logic without the UI). Ideally, you could 
> separate out persistence as well, which means you can run tests of the 
> application without the repository. Greater separation means more 
> opportunity for parallel development (i.e. of views vs interaction 
> model vs persistence) of features.
>
> *Agreement* (Philippe, PJE, Reid, Andi, John, Mikeal): Cleaner 
> separation of UI from the rest of the app.
> *Agreement*: (Philippe, PJE, Andi): Not persisting redundant/constant 
> UI data.
> *Agreement* (John, PJE): The current template mechanism in Blocks is 
> confusing and mostly a historical artifact, so it should be removed.

The CPIA remnants and what Reid described as the "wall of abstraction" 
must go. If we could organize the rearchitecture project so that this 
part was done first and merged with the trunk before the rest (full 
testability, performance and scalability), I'd vote for that.

> Performance, Scalability
> ------------------------
>
> *Open Issue*: (Grant, Andi, Brian K) Unclear what the goals are w.r.t. 
> email beyond what we have today. Need measurements of how Chandler 
> performs in the presence of many items (and/or collections), as well 
> as explicit performance goals.

As an objective, we should aim at handling thousands of  items in 
Chandler: emails, events, tasks, snippets of all kind (see Katie's email 
in that thread). So far though, it doesn't seem to me the most urgent 
issue, as I feel the one listed previously (CPIA architecture obscurity) 
is a road block on any effort. Also, one could think about addressing 
performance and scalability at the repo level, without changing the 
whole architecture.

> Timing of 1.0 Release
> ---------------------
>
> Davor: Shouldn't a "1.0" release happen pretty soon (a few months) 
> after a "0.7"?
> Piero: Not necessarily (depends on what needs to be addressed). 
> Project needs some kind of a roadmap.

I don't think that this "Timing of 1.0" is an issue and it begs the 
question of what 1.0 is. I'd prefer to focus on user adoption and user 
driven development rather than chasing an ill defined 1.0. One thing we 
know though is that our architecture has some rigidity (all listed 
above) that'll make any evolution difficult so addressing those 
rigidities while defining the 1.0 is OK in my opinion.

Cheers,
- Philippe