[pylucene-dev] outOfMemory StandaradAnalyzer

[Andi Vajda]

On Wed, 16 Jan 2008, anurag uniyal wrote:

> In my application I need to create several analyzers on the fly, which 
> leads to outOfMemory problem. see code below

Indeed, you found a leak of local refs to jstring objects when converting a 
Python array into a java array. This is fixed in svn rev 380.

Thanks !

Andi..

----
import lucene
lucene.initVM(lucene.CLASSPATH, maxheap='1m')
stopwords = ["and", "if", "at", "the"]
for i in xrange(100000):
     if i%1000==0: print i
     analyzer = lucene.StandardAnalyzer(stopwords)
     #analyzer = lucene.StandardAnalyzer() # works without stopwords
----

I hope analyzer should be freed?
It works if I do not pass any stopwords to StandardAnalyzer

rgds
Anurag



----- Original Message ----
From: Andi Vajda <vajda at osafoundation.org>
To: pylucene-dev at osafoundation.org
Sent: Monday, 14 January, 2008 11:06:54 PM
Subject: Re: [pylucene-dev] detachCurrentThread crash


On Mon, 14 Jan 2008, anurag uniyal wrote:

> Is it expected to explicity remove any references to local/class variables 
> in threads?

If you create threads faster than Python and Java can collect the objects 
you leave lying around, then yes.

For instance, closing the store you're opening in every thread instead of 
None'ing it out may have the same effect.

Andi..

Otherwise VM crashes. e.g.
-----------
import threading
import lucene
lucene.initVM(lucene.CLASSPATH)
def threadFunc():
     lucene.getVMEnv().attachCurrentThread()
     _store = lucene.FSDirectory.getDirectory("/tmp/index/", True)
     #_store = None # set to None to avoid crash!

     lucene.getVMEnv().detachCurrentThread()
def main():
     t = threading.Thread(target=threadFunc)
     t.start()
     t.join()
main()
----

I need to call detachCurrentThread because I use threads heavily andI will soon get out-of-memory error if didn't call detachCurrentThread.

rgds
Anurag


       Save all your chat conversations. Find them online at http://in.messenger.yahoo.com/webmessengerpromo.php
_______________________________________________
pylucene-dev mailing list
pylucene-dev at osafoundation.org
http://lists.osafoundation.org/mailman/listinfo/pylucene-dev


       Forgot the famous last words? Access your message archive online at http://in.messenger.yahoo.com/webmessengerpromo.php