[Chandler-dev] Pre-loading items as a performance optimization

[Grant Baillie]

On 5 Oct, 2006, at 09:42, Morgen Sagen wrote:

>
> On Oct 4, 2006, at 3:30 PM, Grant Baillie wrote:
>
>> On 4 Oct, 2006, at 14:37, Morgen Sagen wrote:
>>
>>> I did try replacing those mStart and mEnd calls like this:
>>>
>>>    testVal = getattr(EventStamp(view[key]), startAttrName)
>>>
>>> with calls like this:
>>>
>>>    testVal = view.findValue(key, getattr(EventStamp,  
>>> startAttrName).name)
>>>
>>> to avoid loading, but overall a similar number of items was being  
>>> loaded anyway (32 items loaded versus 34, if I recall).  Also,  
>>> there are some cases where findValue( ) doesn't work -- when an  
>>> attribute is Calculated, the item needs to be loaded and getattr  
>>> used.
>>
>> True, but you can refactor the Calculated to work with  
>> view.findValues(). This is hard to do in the general case, but it  
>> turns out that those mStart() and mEnd() functions are only ever  
>> called with a startAttr of 'effectiveStartTime' and endAttr of  
>> 'effectiveEndTime'.
>
> Right -- rather than reimplementing Calculated, I did this:
>
>         try:
>             testVal = view.findValue(key, getattr(EventStamp,  
> startAttrName).name)
>         except AttributeError:
>             # Guess we have to load the item, since findValue can't  
> retrieve a
>             # Calculated
>             testVal = getattr(EventStamp(view[key]), startAttrName)
>
> The exception only happened a few times (for those 'effective'  
> times), but overall there wasn't much change to the timings.

My point is that you can refactor to get rid of the view[key] even in  
the AttributeError case. However, the improvement I was seeing in the  
timings was actually due to a bug on my part, so you're right that it  
doesn't really help.

Looking at the profiles, it looks to me as if compare indexes are  
taking up a big chunk of time. Those necessitate instantiating log(N)  
items on insert, where N is the total number of items in the collection.

It seems to me we could certainly speed up item creation by having a  
type of index that worked like a compare index, but worked on UUIDs.  
Since I already did a bunch of the client-side work to enable that, I  
wouldn't object if Andi implemented one :). It would also enable  
getting rid of the @schema.Comparator hack I added to support compare  
indexes on Stamps, which I never liked in the first place.

--Grant