[pylucene-dev] PyLucene optimize error (due to file size limit?)

[chan kang]

Hi...
I've read a posting<http://lists.osafoundation.org/pipermail/pylucene-dev/2004-August/000089.html>from
2004 about error regarding PyLucene's optimize().
<http://lists.osafoundation.org/pipermail/pylucene-dev/2004-August/000089.html>
at the end of the thread, he said that he's solved the problem by employing

"gcc (GCC) 3.5.0 20040717".

I thought my situation is somewhat similar to his.
What happens is : my optimize() function ends up with JavaError.
It doesn't even say where the error is made...
The error message is as follows:

optimizing................
> Traceback (most recent call last):
> File "in.py", line 124, in ?
> writer.optimize()
> File "/usr/lib/python2.4/site-packages/PyLucene.py", line 2276, in
> optimize
> def optimize(*args): return _PyLucene.IndexWriter_optimize(*args)
> PyLucene.JavaErrorhola:/usr/lib/cgi-bin#




Since I've been encountering this type of error ever since I tried to
optimize over
larger indices, I've switched my approach to NOT optimizing only once at the
end,
but optimizing at the end of certain amount of transactions(writing to
index).
Whenever the optimizing is successfully carried out, I copy it to another
temporary directory,
so the resulting copied index is always the successfully optimized version.

However, I'm left with the same type of error.
So I've looked into the index directory, and found out that the
"so-far-successful index" is
2GB. I think this means that the optimize() was successful until 2GB, and
the error occurred when
I tried to optimize the index to create a single file with more than 2GB.
So I guessed that it might be the file size limitation for linux, because
the figure is exactly "2gb",
and tried to make a new file of more than 12GB, by using the command
dd if=/dev/zero of=big.file bs=1M count=12000


To my surprise, the 12GB file was successfully created... Does this mean
that it has nothing to do with the linux file size limit?
My gcc version is 4.x (gcc (GCC) 4.0.3 20060212 (prerelease) (Debian 4.0.2-9)).
, file system ext3, and kernel version is debian 2.6.15
After reading the
posting<http://lists.osafoundation.org/pipermail/pylucene-dev/2004-August/000092.html>,
I started to seriously consider about actually "downgrading" to gcc version
3.5.0,
because he said that worked...

What could be the problem?

Please help... If this problem is not solved, i might not be able to use
PyLucene at all, because the data I'm dealing with is pretty large.


Thank you in advance.
<http://lists.osafoundation.org/pipermail/pylucene-dev/2004-August/000089.html>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.osafoundation.org/pipermail/pylucene-dev/attachments/20060318/d883f839/attachment.htm