[pylucene-dev] GC Warning: Out of Memory! Returning NIL!
info at altervision.biz
Wed Feb 16 11:18:48 PST 2005
Well, the problem is not about just IndexSearcher on big index. The
problem is about many PythonThreads.
I can open index 20gb or 50gb with IndexSearcher without problem.
But when I try to create many PythonThreads which operates with
IndexSearcher opened on big index then I receive exceptions like these
GC Warning: Header allocation failed: Dropping block.
GC Warning: Out of Memory! Returning NIL!
And when I have decreased number of threads to 2 or 3 threads then
this error has gone.
So, the question is: How can PythonThread affect to this? Why the less
amount of them does not produce exceptions of this kind?
>> I have following structure of program:
>> I am trying to create 10 running threads of LuceneWorkerThread from
>> main thread.
>> When I set number of threads to 4 or less then it runs without
>> exception. Number of PythonThreads affect to this exception!
>> Any ideas?
AV> I don't know what the memory requirements are for a given index size. That is
AV> a question for the lucene-user or lucene-dev mailing lists. If you reach the
AV> libgcj memory limits, there may be some environment variables you can set to
AV> change them too. That is a question for the java at gcc.gnu.org mailing list. I
AV> do not know what they are either, but I've seen them posted and discussed
AV> before there.
AV> Here is what I'd do resolve this issue :
AV> 1. Try it under Java, if it fails the same way, solve it there first, with
AV> help from the lucene mailing lists. To open up a huge index with Java
AV> Lucene, you actually don't need to write a single line of java code, the
AV> src/demo/org/apache/lucene/demo/SearchFiles.java should be enough, it
AV> takes a command line argument to an index directory.
AV> If there are patches that can be applied to the Java Lucene code base
AV> before it gets compiled into PyLucene, that help with this, I'd be happy
AV> to apply them.
AV> 2. If it works fine under Java yet fails under python/PyLucene, investigate
AV> what the libgcj memory limits are, with help from the java at gcc.gnu.org
AV> mailing list.
AV> 3. If all fails then we have a harder problem that will require some
AV> rethinking and some fixing of the underlying software packages used.
AV> By the way, you say that you have a 51 Gb index and a 20 Gb index. What is the
AV> size of the biggest single index files in these index directories ? There used
AV> to be a bug in libgcj that it couldn't support files bigger than 2 or 4 Gb (I
AV> don't remember). I know this bug is fixed in gcj 4.0, a PyLucene user actually
AV> verified that. I do not know that this bug has been fixed in the version of
AV> gcj we're using, gcj 3.4.3.
More information about the pylucene-dev