[pylucene-dev] two different JCC modules in same VM; initVM()?

[Andi Vajda]

On Mon, 10 Mar 2008, Andi Vajda wrote:

> I got the multi-jcc-built-extension-in-same-process thing to work. There are 
> good news and bad news:
>
> - the good:
>   . it works :)
>
> - the bad:
>   . classpath is set once per VM, there may be some class load tricks
>     that can be played but it's not something on the radar at the moment.
>   . it requires some JCC runtime code to be included in a shared library
>     instead of a python extension (on Mac, a .dylib instead of an .so
>     bundle).
>
> Python's distutils/setuptools is not so good at knowing how to build shared 
> libraries. I talked with the person behind setuptools and he said that the 
> trunk had code to support this but he wasn't sure in what state it was in. He 
> also said that patches to this were welcome. I'm going to look at this as 
> I've hit this problem before. Supporting shared libraries between python 
> extension bundles is a good thing to have in setuptools.
>
> I think that I'm going to check-in something soon where the shared library 
> building pleasure is left to the user ;) but not making it any harder to 
> build the current, default way where the JCC runtime is statically linked 
> into the extension. Later, once I figure out the setuptools thing, I can 
> update jcc's setup.py to also build the shared library.

I just checked in the changes to support this properly.

initVM() no longer takes an env argument but still needs to be called for 
every extension being imported into the process. There is only one env per 
process and any extension can return it. The python JCCEnv type is defined 
in each extension module but from the _same_ static structure in the shared 
library and Python only initializes it once.

Sadly, the classpath additions don't seem to stick so it must be set either 
in the environment or via the classpath keyword to initVM().

I made it an error to pass in any other arguments to initVM() on any calls 
other than the first one since these would be ignored. In other words, you 
get one chance per process to start and configure the Java VM.

To try this out you need:

    - a libjcc.dylib shared library
      To build this, simply enter this in the shell after building JCC
      itself (on leopard intel mac, adapt for other platforms until I
      automate this in JCC's setup.py)

g++ -dynamiclib -install_name /Library/Python/2.5/site-packages/JCC-1.8-py2.5-macosx-10.5-i386.egg/jcc/libjcc.dylib -o libjcc.dylib build/temp.macosx-10.5-i386-2.5/jcc/sources/jcc.o build/temp.macosx-10.5-i386-2.5/jcc/sources/JCCEnv.o -undefined dynamic_lookup
cp -p libjcc.dylib /Library/Python/2.5/site-packages/JCC-1.8-py2.5-macosx-10.5-i386.egg/jcc/

      This reuses the .o files built by setup.py (they're compatible). The
      g++ statement just links two files together into libjcc.dylib in a way
      that forces using libraries to look for it in the installed jcc egg.
      You can, of course, change -install_name to suit your install or omit
      it altogether and set DYLD_LIBRARY_PATH instead (not so good for
      security).

    - build all your extensions by invoking jcc with --shared (the default is
      unchanged, shared is False)
      This tells jcc to link with libjcc.dylib instead of statically
      linking with the same JCC runtime code. Again, libjcc.dylib is expected
      to be installed into the JCC egg dir (manually).

Then, once you've got everything built and installed, call initVM() on each 
of your extension modules, passing in all relevant info into the first call 
as follows:

   >>> import os, lucene, gs
   >>> lucene.initVM(os.pathsep.join([lucene.CLASSPATH, gs.CLASSPATH]),
                     other args as needed)
   >>> gs.initVM()
   >>> lucene.Document()
   >>> gs.GoodStuff.Frob()


Andi..