[Dev] Blocks and Queries

[Ted Leung]

This note is a summary of a phone call that John and I had about 
queries and blocks.  John, please let me know that I captured 
everything correctly.  Everyone else can pile on as needed/interested.

John has an implementation of the way that blocks would like to use 
queries. This implementation can be found in
osaf/chandler/Chandler/parcels/OSAF/framework/blocks/parcel.xml
osaf/chandler/Chandler/parcels/OSAF/framework/blocks/Query.py

In the application, blocks have queries associated with them.  The 
blocks know which queries they are using, and have access to the result 
set of the queries.  All interaction with the repository is via these 
queries, which are stored as items in the repository.  A block will 
rely on notification/observable queries to load/update repository data 
for display.  There shouldn't be any code in a block that explicitly 
runs a query.  It was good for me to get a deeper understanding of how 
the application is going to use/rely on queries.

The key services that the application needs from queries/the repository 
are:

1. The application needs indexable (random access with good 
performance) cached query results.
The UI needs to be able to scroll to various points in the query 
result, or jump from one portion to another.  The current 
implementation of reference collections provides a list style API, and 
is implemented as a doubly linked list.  This data structure is not 
suited for random access.  There is a dict style of reference 
collection, but Andi has stated that this is deprecated, although dict 
based code appears to run at the moment.

2. The application needs needs good hints about the size of query 
results.
In order to do a good job in the UI, the application needs to know the 
size of the query result.  The size information is a hint, and may be 
updated in the case of incremental query evaluation or a query running 
in a thread.  A scenario like the following is acceptable:

Add 50 items to the result set, update the result size hint, notify the 
application that the result set has changed (so it can redraw)
Add another 50 items to the result set, update the result size hint, 
notify the application that the result set has changed (so it can 
redraw)
And so on.

3. Delayed loading of items is important -- this is important for a 
responsive UI.
We are probably moving the direction of running all queries (and other 
repository operations) in a thread(s).
This likely has profound implications for our current threading model, 
since threads do not share views of the repository.  In order for two 
thread to share, they will have to commit items.  Forcing threads to 
commit is a problem because the undo model of the application is based 
on commit.

4. Notification/Observable queries
This was probably the area which was most surprising (although not 
necessarily in a bad way).  Up until this conversation, I was under the 
impression that notification was a "nice to have" feature.  After the 
call, I realize that this is not the case, as notification is a key 
part of the Query/Block API.   I need to sit down and do a little more 
thinking and updating of the design based on this.  I probably also 
need to look more deeply at notification stuff in general.

John pointed out two issues:

a. Persistent notification registrations.
If blocks are going to use notification as a way of updating their 
content, then notification registrations will probably need to be 
persistent, which is not how they are now.  Speaking from ignorance, I 
think that this should be relatively easy to do as a layer over the 
current notification architecture.

b. Efficient notification
John was concerned the we not notify queries unless absolutely 
necessary.  He gave the example that only blocks which are being 
displayed need to be notified when they query result changes.  There is 
probably a little more to this than meets the eye, otherwise updates 
will get lost and queries may need to get re-run.  In any case, it's 
clear that someone needs to think about this some more (at least more 
than I have).

5. Where is the line between Queries and the Application?
This is a little bit hard to define right now.  We probably want the 
queries used by a block to be a behavioral subclass of the QueryItem 
that we end up with from the repository.  This is more a question of 
layering, and probably doesn't need to be answered at the moment.  
Since John has an API that is in use, we can plan for how to change 
that API (or migrate off of that should it be necessary -- although I 
can't think of any reason that it would be).

----
Ted Leung                 Open Source Applications Foundation (OSAF)
PGP Fingerprint: 1003 7870 251F FA71 A59A  CEE3 BEBA 2B87 F5FC 4B42