[pylucene-dev] GC Warning: Out of Memory! Returning NIL!

[Yura Smolsky]

Hello, Andi.

>> I can open index 20gb or 50gb with IndexSearcher without problem.
AV> That's not what you said yesterday.

Im sorry, that I have messed things.

>> And when I have decreased number of threads to 2 or 3 threads then
>> this error has gone.
>> So, the question is: How can PythonThread affect to this? Why the less
>> amount of them does not produce exceptions of this kind?

AV> There may be overhead involved in having multiple threads against a given
AV> index. Have you tried this under Java yet ?

No b/c I thought that its a problem of gcj's gc...

AV> By the way, you say that you have a 51 Gb index and a 20 Gb index. What is the
AV> size of the biggest single index files in these index directories ? There used
AV> to be a bug in libgcj that it couldn't support files bigger than 2 or 4 Gb (I
AV> don't remember). I know this bug is fixed in gcj 4.0, a PyLucene user actually
AV> verified that. I do not know that this bug has been fixed in the version of
AV> gcj we're using, gcj 3.4.3.

total 51183104
-rw-rw-r--    1 icerocket icerocket 11938376 Feb  2 12:13 _3z3mu.f1
-rw-rw-r--    1 icerocket icerocket 11938376 Feb  2 12:13 _3z3mu.f2
-rw-rw-r--    1 icerocket icerocket 11938376 Feb  2 12:13 _3z3mu.f3
-rw-rw-r--    1 icerocket icerocket 11938376 Feb  2 12:13 _3z3mu.f4
-rw-rw-r--    1 icerocket icerocket 11938376 Feb  2 12:13 _3z3mu.f5
-rw-rw-r--    1 icerocket icerocket 11938376 Feb  2 12:13 _3z3mu.f6
-rw-rw-r--    1 icerocket icerocket 11938376 Feb  2 12:14 _3z3mu.f7
-rw-rw-r--    1 icerocket icerocket 11938376 Feb  2 12:14 _3z3mu.f8
-rw-rw-r--    1 icerocket icerocket 14437452892 Feb  2 09:53 _3z3mu.fdt
-rw-rw-r--    1 icerocket icerocket 95507008 Feb  2 09:56 _3z3mu.fdx
-rw-rw-r--    1 icerocket icerocket       69 Feb  2 06:29 _3z3mu.fnm
-rw-rw-r--    1 icerocket icerocket 4312089378 Feb  2 11:03 _3z3mu.frq
-rw-rw-r--    1 icerocket icerocket 4747749584 Feb  2 12:05 _3z3mu.prx
-rw-rw-r--    1 icerocket icerocket  7632774 Feb  2 12:13 _3z3mu.tii
-rw-rw-r--    1 icerocket icerocket 524378115 Feb  2 12:13 _3z3mu.tis

I have resolved 2Gb limit problem much time ago :)

AV> Also, I just learned that using an unoptimized index is going to require more
AV> memory. How much more is a question for lucene-user as well. Optimizing your
AV> index is likely to push you over the 4Gb per file limit in gcj < 4.0 though.
AV> Have you tried it ? (backing up your existing index first).

My index is created using merge factor of 2. They are almost optimized
:)

Yura Smolsky.