[pylucene-dev] finalizing the deadly embrace

[Andi Vajda]

On Tue, 22 Jan 2008, anurag uniyal wrote:

> Here are the results of dumprefs for few failed runs
> (calling finalize for filter after every invocation do solves the problem)

Ok, so you still have a workaround. I'm about to go offline for two weeks 
and intend to get back to this upon my return.

> My observation: It happens at different times but for 1m heap memory, it 
> happens when MyFilter count (which i keep tracking using __del__) reaches 
> about 49.
> dumpRefs also shows almost same count for LowerCaseFilter class but 
> MyFilter/PythonTokenStream is not listed there?
> Also in first case there is no filter class at all in dumpRefs output?

Another thing that could help - in theory - is calling gc.collect() and 
System.gc() periodically.

Andi..

class org.apache.lucene.index.IndexReader$FieldOption 10
class [Ljava.lang.String; 10
class java.lang.String 11
class java.util.Locale 21
class java.lang.Class 114
runCount=602 totalRefs 244

class [Ljava.lang.String; 10
class java.lang.String 11
class java.util.Locale 21
class org.apache.lucene.analysis.LowerCaseFilter 48
class java.lang.Class 114
runCount=307 totalRefs 291

class [Ljava.lang.String; 10
class java.lang.String 11
class java.util.Locale 21
class org.apache.lucene.analysis.LowerCaseFilter 49
class java.lang.Class 114
runCount=497 totalRefs 292

class [Ljava.lang.String; 10
class java.lang.String 11
class java.util.Locale 21
class org.apache.lucene.analysis.LowerCaseFilter 49
class java.lang.Class 114
runCount=557 totalRefs 292

rgds
Anurag


----- Original Message ----
From: Andi Vajda <vajda at osafoundation.org>
To: anurag uniyal <anuraguniyal at yahoo.com>
Cc: pylucene-dev at osafoundation.org
Sent: Tuesday, 22 January, 2008 12:49:01 AM
Subject: Re: [pylucene-dev] finalizing the deadly embrace


On Mon, 21 Jan 2008, anurag uniyal wrote:

> This still fails *here* after removing __del__

> I did rebuild JCC thats why now custom analyzer/parser works without 
> finalize.
> test_PythonDirectory.py is also passing.

When run with -loop ?

> src : trunk rev 381
> lucene.VERSION = 2.2.0-613493
> Python 2.4.3
> Ubuntu 6.06.1 LTS
> java 1.5.0_06-b05

Ok, I have such a system available to me, I'm going to give your test a try 
there.

Still, could you in the meantime try to find out what is leaking ?
If Java is running out of memory, one way to track down what is causing it 
is to look at the global refs PyLucene keeps to Java objects.

First, after each test cycle, print out the size of env._dumpRefs().
env is the return value from initVM()

   print len(env._dumpRefs)

if that size is growing, next thing is to get the classes of these objects:

   refs = sorted(env._dumpRefs(classes=True).items(),
                 key=lambda x: x[1], reverse=True)
   print refs[0:4]

This would print out the classes of the 5 most important Java object leaks 
via global refs and their counts. (the code above is used in 
test/test_Sort.py, for example).

Andi..


       Save all your chat conversations. Find them online at http://in..messenger.yahoo.com/webmessengerpromo.php