[pylucene-dev] memory leak status

Andi Vajda vajda at osafoundation.org
Thu Jan 10 12:15:36 PST 2008

In the past few days, several users reported symptoms of memory leaks in 
jcc-PyLucene. After a bit of sleuthing, I found two leaks:

    1. A reference to a Python Java wrapper was leaked whenever an inherited
       method was called on the Java object from Python (in callSuper).
       Fixing this one was trivial and is checked in to the svn trunk.

       I don't know if this fixes the cases reported but it certainly has
       a major impact. For instance, any time searcher.search(query) is
       called, the searcher is leaked (!).

       To verify this, run:
       > python test/test_PyLucene.py Test_PyLuceneWithFSStore.test_searchDocuments -loop
       Without the fix, after a short while, the VM runs out of memory.
       With the fix, it seems to be running forever (and speed remains more or
       less constant)

    2. A "deadly embrace" between a Python extension instance and its Java
       parent instance is currently preventing Python extention instances
       and their Java parent from ever being freed. The Python extension
       instance is holding a reference to the Java parent instance and the
       Java parent instance is holding a reference to the Python extension
       Without some explicit intervention, this cross-VM cycle can't be
       broken. I'm currently thinking of making it possible to call finalize()
       on these objects manually to break the cycle. I'm also thinking of
       adding a GC thread to the process that would garbage collect the
       extension instances with no more than two counted python refs. This
       thread, combined with weak global refs on the JNI side could make
       collecting these Python extension instances semi-automatic. Needless
       to say, I don't like this idea too much and I'm trying to find another
       less complicated way. In the meantime, I think I'm going to be adding
       support for the manual way via finalize() shortly.

       This leak (still in svn trunk), is not normally that bad, as Python
       extension instances are rarer (than the earlier leak) and leaking them
       is, normally, not as deadly. Still, there are cases where it is bad as
       when implementing a Python extension for Directory and its sibling
       To see for yourself, try running test/test_PythonDirectory.py in a
       loop. This leak is a problem in the current Chandler release, for
       example, where such a Python extension is used.

       More on this leak in the next few days.

In order to debug these leaks, I improved env._dumpRefs() a bit by adding 
some keywords to it.

_dumpRefs() now can be called in three ways:

    - _dumpRefs(): returns a list of tuples (system hash id, ref count)
      these are useful for quickly getting an idea of how many global Java
      referenced objects there are at the moment (these objects are not GC'ed
      by Java until removed from the refs table)

    - _dumpRefs(classes=True): returns a dict of { className: instance count }
      to get an idea about how many instances of various classes are being
      thus kept from being GC'ed by Java

    - _dumpRefs(values=True): returns a list of tuples (value string, ref
      count) to get an idea of what the values look like. This is to be used
      with caution a printing out Java values can be expensive.

It would be interesting to see if the people who recently reported memory 
leak symptoms on this list could try the current trunk and report if that 
solves their problem or at least, improves on it.
If you are re-building PyLucene from the trunk to try this out, be sure to 
completely rebuild jcc first (the fix is there).

Thank you for your patience !


More information about the pylucene-dev mailing list