[pylucene-dev] outOfMemory StandaradAnalyzer

[anurag uniyal]

In my application I need to create several analyzers on the fly, which leads to outOfMemory problem. see code below
----
import lucene
lucene.initVM(lucene.CLASSPATH, maxheap='1m')
stopwords = ["and", "if", "at", "the"]
for i in xrange(100000):
    if i%1000==0: print i
    analyzer = lucene.StandardAnalyzer(stopwords)
    #analyzer = lucene.StandardAnalyzer() # works without stopwords
----

I hope analyzer should be freed?
It works if I do not pass any stopwords to StandardAnalyzer

rgds
Anurag



----- Original Message ----
From: Andi Vajda <vajda at osafoundation.org>
To: pylucene-dev at osafoundation.org
Sent: Monday, 14 January, 2008 11:06:54 PM
Subject: Re: [pylucene-dev] detachCurrentThread crash


On Mon, 14 Jan 2008, anurag uniyal wrote:

> Is it expected to explicity remove any references to local/class variables 
> in threads?

If you create threads faster than Python and Java can collect the objects 
you leave lying around, then yes.

For instance, closing the store you're opening in every thread instead of 
None'ing it out may have the same effect.

Andi..

Otherwise VM crashes. e.g.
-----------
import threading
import lucene
lucene.initVM(lucene.CLASSPATH)
def threadFunc():
    lucene.getVMEnv().attachCurrentThread()
    _store = lucene.FSDirectory.getDirectory("/tmp/index/", True)
    #_store = None # set to None to avoid crash!

    lucene.getVMEnv().detachCurrentThread()
def main():
    t = threading.Thread(target=threadFunc)
    t.start()
    t.join()
main()
----

I need to call detachCurrentThread because I use threads heavily andI will soon get out-of-memory error if didn't call detachCurrentThread.

rgds
Anurag


      Save all your chat conversations. Find them online at http://in.messenger.yahoo.com/webmessengerpromo.php
_______________________________________________
pylucene-dev mailing list
pylucene-dev at osafoundation.org
http://lists.osafoundation.org/mailman/listinfo/pylucene-dev


      Forgot the famous last words? Access your message archive online at http://in.messenger.yahoo.com/webmessengerpromo.php
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.osafoundation.org/pipermail/pylucene-dev/attachments/20080116/a24e47fa/attachment.htm