[pylucene-dev] Re: customizing java classes in python -- performance?

[Andi Vajda]

On Thu, 18 May 2006, Mike Klaas wrote:

> Am I asking in the wrong place, or has no-one really dug into these issues?

I don't know that anyone has run timings of these exact operations.
Since PyLucene got de-swigged, crossing the python<->CNI barrier involves:

   python -> C++/Java
     - parsing calling args, maybe
     - releasing the GIL
     - calling C++
     - acquiring the GIL
     - packaging up return values, maybe

   C++/Java -> python
     - acquiring the GIL
     - creating a calling args tuple, maybe
     - calling Python
     - extracting return values, maybe
     - releasing the GIL

I don't think that calling python from C++ is any slower than calling python 
from python. It seems that the GIL operations are pretty fast unless there is 
contention.

Getting rid of the GIL operations, even in carefully selected places, is a 
recipe for deadlocks or crashes. Keep in mind that the libgcj/gc finalizer 
thread acquires the GIL when it calls into Python for releasing wrapped Python 
object references.

If you want speed over anything, write the Java class extensions in Java with 
extension methods declared native, and provide the native C++ methods via CNI.
If you prefer convenience and the usual python speed is enough, use the 
built-in PyLucene extension points.

If you do actual timings, let us know. Thanks !

Andi..

>
> cheers,
> -MIke
>
> On 5/15/06, Mike Klaas <mike.klaas at gmail.com> wrote:
>> I'm currently investigating using PyLucene for a large scale
>> distributed search application which is currently in 95% python with a
>> smattering of c/pyrex for performance.  I'm rather impressed with the
>> package, particularly the ability to "subclass" certain lucene classes
>> to provide customization hooks.
>> 
>> I was curious if anyone has done heavy-duty lifting using these
>> reverse bindings.  In particulary, I am concerned with subclasses
>> which are tightly coupled with the core lucene query code -- like
>> SortComparator.  Does the repeated python<->CNI boundary crossing
>> degrade performance unacceptably?  Has anyone had success implementing
>> these sorts of customizations java-side, recompiling the whole package
>> with gcj?
>> 
>> Thanks in advance,
>> -MIke Klaas
>> 
>