[Chandler-dev] item reference rework (perf, ram)

Andi Vajda vajda at osafoundation.org
Tue Jan 9 22:22:01 PST 2007


I just checked-in a complete rework of item references. SingleRef, the simple 
UUID wrapper type is gone and replaced by a smarter type, ItemRef. Similarly, 
the issingleref() function was replaced by isitemref().

ItemRef wraps a UUID, a repository view and a weak (non-counted) reference to 
an Item instance. Each Item instance in every view has a corresponding unique 
ItemRef instance. It is safe to use the 'is' operator to test ItemRef 
equality.

Because an ItemRef caches a reference to the actual Item instance, 
de-referencing it is considerably faster than finding the item behind a 
SingleRef. Once an Item instance is unloaded, de-referencing the ItemRef 
causes the Item instance to be reloaded automatically.

To de-reference an ItemRef, call it (as you would a regular python weak ref):
   >>> ref = item.itsRef
   >>> ref() is item
   >>> True

Because an ItemRef is unique per Item instance per view, re-loading the Item 
instance is done only once, all references to the unique ItemRef now have an 
Item instance one de-reference away. SingleRefs were not unique and 
de-referencing them always involved calling view.find().

Because the Item instance cached by an ItemRef is not ref-counted (it's a weak 
reference), Python will unload and free Item instances much more readily. No 
items are pinned by default anymore. Item pruning (unloading of the least 
recently used Item instances from a view) is now applied to any Item that has 
no hard references outside of the repository code.

I experimented with the new --prune command line option which sets a default 
view size (loaded item count) that the repository will try to prune to after 
each refresh and commit and it seems that a vanilla chandler main view likes 
to have around 1,500 items loaded.

On the other hand, other views can be pruned down to much much smaller item 
counts.

Keeping a view small reduces the amount of memory used by Chandler. It doesn't 
look like Python is returning memory to the OS (correct me if I'm wrong) but 
it sure is reusing memory that is freed by pruning Item instances from a view.

Using a prune size that is too small would causes items to be unloaded and 
reloaded constantly, slowing down operations significantly. Choosing the 
correct prune size for each view requires some fine tuning by their 
developers.

Andi..



More information about the chandler-dev mailing list