[pylucene-dev] Re: OT: what's wrong with bsddb?

[Andi Vajda]

Sorry, that URL to the docs about MVCC again:
http://www.oracle.com/technology/documentation/berkeley-db/db/ref/transapp/read.html

Andi..

On Fri, 5 Jan 2007, Andi Vajda wrote:

>
> On Fri, 5 Jan 2007, Peter Fein wrote:
>
>> On Friday 05 January 2007 12:31, Andi Vajda wrote:
>>> On Thu, 4 Jan 2007, Peter Fein wrote:
>>>> [0] Has anyone else had difficulties w/ bsddb (outside PyLucene)?
>>>> Despite it's excellent reputation, it's given us lots of trouble.  We've
>>>> been using gdbm instead.
>>> 
>>> What kind of troubles ?
>> 
>> Hmm, let's see:
>> 
>> * len(db) doesn't work reliably - sometimes gives wrong/negative answers
>> * keys sometimes get 'lost' when iterating over largish (~200K) db's - the
>> data's not corrupted, the library just confuses itself.  There's a python 
>> bug
>> for this, I can't find it ATM.
>
> I see, bugs with the bsddb python API wrappers. I don't use that API it 
> anymore as it's usually out of date with Berkeley DB itself since it's not 
> maintained by Sleepycat (now Oracle) but is part of Python itself. Because 
> these two software products don't coordinate their release cycles, this 
> invariably leads to the APIs and their wrappers to be out of sync. I ended up 
> writing my own python wrappers for Berkeley DB giving me much finer control 
> over how data was accessed and better choice of API usage [1].
>
>> * DBNoMemoryError: (12, 'Cannot allocate memory -- Lock table is out of
>> available locker entries') - this was the last straw, never found out what
>> the problem was.  Some sort of create-time config option, IIRC.
>
> Yes, I saw this a long time ago and worked it around by adding more such 
> lockers with the set_lk_max_locks() and set_lk_max_objects() APIs.
> The new Berkeley DB 4.5.20 release supports MVCC [2][3] and that more or less 
> does away with needing read locks. If your usage patterns fit the MVCC 
> constraints, it's a very neat feature.
>
>> Basically, we found it just required too much hand-holding / broke far too
>> easily to make it worthwhile.
>
> Yes, it can be finnicky but I've found it quite good if handled correctly.
> Handling locks and deadlock situations can be particularly troublesome but 
> again, understanding your access patterns helps a lot with untangling the the 
> locks.
>
> Andi..
>
> [1] 
> http://svn.osafoundation.org/chandler/trunk/internal/chandlerdb/chandlerdb/persistence/
> [2] 
> http://www.oracle.com/technology/documentation/berkeley-db/db/api_c/frame.html
> [3] 
> http://lists.osafoundation.org/pipermail/chandler-dev/2007-January/007481.html
> _______________________________________________
> pylucene-dev mailing list
> pylucene-dev at osafoundation.org
> http://lists.osafoundation.org/mailman/listinfo/pylucene-dev
>