[pylucene-dev] GC Warning: Out of Memory! Returning NIL!

[Yura Smolsky]

Hello, Andi.

Well, the problem is not about just IndexSearcher on big index. The
problem is about many PythonThreads.

I can open index 20gb or 50gb with IndexSearcher without problem.
But when I try to create many PythonThreads which operates with
IndexSearcher opened on big index then I receive exceptions like these
GC Warning: Header allocation failed: Dropping block.
GC Warning: Out of Memory!  Returning NIL!

And when I have decreased number of threads to 2 or 3 threads then
this error has gone.
So, the question is: How can PythonThread affect to this? Why the less
amount of them does not produce exceptions of this kind?

>> I have following structure of program:
>>
>> I am trying to create 10 running threads of LuceneWorkerThread from
>> main thread.
>>
>> When I set number of threads to 4 or less then it runs without
>> exception. Number of PythonThreads affect to this exception!
>>
>> Any ideas?

AV> I don't know what the memory requirements are for a given index size. That is
AV> a question for the lucene-user or lucene-dev mailing lists. If you reach the
AV> libgcj memory limits, there may be some environment variables you can set to
AV> change them too. That is a question for the java at gcc.gnu.org mailing list. I
AV> do not know what they are either, but I've seen them posted and discussed
AV> before there.

AV> Here is what I'd do resolve this issue :

AV>    1. Try it under Java, if it fails the same way, solve it there first, with
AV>       help from the lucene mailing lists. To open up a huge index with Java
AV>       Lucene, you actually don't need to write a single line of java code, the
AV>       src/demo/org/apache/lucene/demo/SearchFiles.java should be enough, it
AV>       takes a command line argument to an index directory.
AV>       If there are patches that can be applied to the Java Lucene code base
AV>       before it gets compiled into PyLucene, that help with this, I'd be happy
AV>       to apply them.

AV>    2. If it works fine under Java yet fails under python/PyLucene, investigate
AV>       what the libgcj memory limits are, with help from the java at gcc.gnu.org
AV>       mailing list.

AV>    3. If all fails then we have a harder problem that will require some
AV>       rethinking and some fixing of the underlying software packages used.

AV> By the way, you say that you have a 51 Gb index and a 20 Gb index. What is the
AV> size of the biggest single index files in these index directories ? There used
AV> to be a bug in libgcj that it couldn't support files bigger than 2 or 4 Gb (I
AV> don't remember). I know this bug is fixed in gcj 4.0, a PyLucene user actually
AV> verified that. I do not know that this bug has been fixed in the version of
AV> gcj we're using, gcj 3.4.3.

AV> Andi..




Yura Smolsky.