[Chandler-dev] new --mvcc performance feature (more) (fwd)

[Andi Vajda]

Reposted with permission :)

Andi..

---------- Forwarded message ----------
Date: Wed, 03 Jan 2007 19:48:48 -0500
From: Andre Mueninghoff <andre_mueninghoff at fastmail.fm>
To: Andi Vajda <vajda at osafoundation.org>
Subject: Re: [Chandler-dev] new --mvcc performance feature (more)

Hi Andi,

Wow, it's like a whole new and different application! Pretty much all
functions seem faster, many of them MUCH faster. A sync of my nine
collections (170+ items total) went from over 5 minutes to less than
one. Amazing. (Hmm, maybe if I use more than nine collections now, I
won't have to restart the restore of the 10th collection and upwards
after they time out.)

Hope you don't mind a direct email...I didn't want to clutter up the dev
list with fan mail. I won't (I hope) always be futzing around with
Chander in my evenings, but I happened to have seen your updates
yesterday evening and this evening (EDT), and had to give it a spin.

Enjoy, Andre

On Wed, 3 Jan 2007 14:09:36 -0800 (PST), "Andi Vajda"
<vajda at osafoundation.org> said:
>
> I forgot to say in my previous message about --mvcc that to use this new
> feature, a new version of chandlerdb is necessary, version 0.6-69.
>
> Use 'make chandlerdb' to upgrade after your next 'svn update'.
>
> Andi..
>
> ---------- Forwarded message ----------
>
> In version 4.5.20, Sleepycat (now Oracle), added support for MultiVersion
> Concurrency Control (MVCC) to Berkeley DB.
>
> Using MVCC causes commit() to make a copy of the pages it modifies
> thereby
> allowing database readers such as cursors to operate without acquiring
> read
> locks. This has three consequences:
>
>    - no more deadlocks:
>
>      even though chandler (Berkeley DB really) detects deadlocks and
>      terminates
>      the transaction of the writer causing the deadlock, not having the
>      deadlock in the first place makes commit ultimately faster since it
>      doesn't have to be retried until no deadlock happens. The best way
>      to
>      experiment with that is using the 'Test -> Restore Settings' menu
>      option
>      and use a .ini file that lists several collections. Chandler.log
>      should
>      not have any 'commit aborted by deadlock' messages and the process
>      should
>      complete much faster (assuming the server used delivers on its
>      side).
>
>    - better read throughput:
>
>      item loading, cursor-based iterations such as history, kind queries,
>      etc... are faster since they don't have to acquire (and wait) for
>      read locks.
>
>    - more memory (cache) usage:
>
>      because commit() has to copy some the pages it modifies, more cache
>      is
>      used by an chandler process using --mvcc.
>
> MVCC should be well suited for Chandler's use of Berkeley DB. Chandler's
> repository is versioned and old versions of data are not modified.
> Instead, new
> versions of data are written. This limits the contention for existing
> pages
> between readers and writers as well. Therefore, doing away with having to
> acquire read locks should be a performance boost.
>
> See [1] for more information about MVCC and so-called "Snapshot
> Isolation".
>
> Because this is a new feature in Berkeley DB, I felt it was safer to make
> it
> experimental and turned on via a new --mvcc command line argument (off by
> default).
>
> It would be good to turn this on on the performance tinderboxes as it's
> having a noticeable effect on performance. Also, if it proves stable (as
> it
> apparently is), I'd like to make it the default in a few weeks...
>
> Andi..
>
> [1]
> http://www.oracle.com/technology/documentation/berkeley-db/db/api_c/frame.html
> _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
>
> Open Source Applications Foundation "chandler-dev" mailing list
> http://lists.osafoundation.org/mailman/listinfo/chandler-dev
> _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
>
> Open Source Applications Foundation "chandler-dev" mailing list
> http://lists.osafoundation.org/mailman/listinfo/chandler-dev
-- 
   Andre Mueninghoff
   andre_mueninghoff at fastmail.fm