[pylucene-dev] pylucene and 2gb limit of files

Yura Smolsky info at altervisionmedia.com
Thu Aug 10 00:25:05 PDT 2006

Hello, Andi.

AV> On Wed, 9 Aug 2006, Yura Smolsky wrote:

>> check it
>> AV> I believe I've seen this too. Patches are welcome :)

AV> Thanks, Yura, it seems to make sense. On Windows, for 'ren' to work, a file
AV> can't exist under the new name whereas on Unix, mv overwrites the target if it
AV> exists.

yes, thats what python manuals tells.

I am going to implement fully functional python directory same to Java
Lucene one. I will provide it here later, so you can include into PyLucene
if somebody will need.

Also, can I contribute pool of PythonThread classes into PyLucene? This class solves
problem (memory leak) of multiple recreation of PythonThread classes.
I think somebody asked same question here before. I have attached

OMG. I jave just tested it on windows and linux with new PyLucene
2.0.1. I see that we have mem leaks on windows! We have not them
before on 1.4.3 - I am sure. And we have not them on linux on 2.0.1
atm. Andi, what could it be?


Yura Smolsky,
-------------- next part --------------

from PyLucene import *
from Queue import *
import time

class _WorkerPythonThread(PythonThread):
    def __init__(self, funFinished, name=None):
        PythonThread.__init__(self, name=name)
        self.__target = False
        self.__args = ()
        self.__kwargs = {}
        self.__funFinished = funFinished
    def setTargetArgs(self, target, args=(), kwargs={}):
        self.__target = target
        self.__args = args
        self.__kwargs = kwargs
    def run(self):
            if self.__target:
                self.__target(*self.__args, **self.__kwargs)
                raise RuntimeError("Target function is wrong")
class PoolPythonThread:
    def __init__(self, amount=10):
        self.__queue = Queue()
        for i in range(amount):
            self.__queue.put(_WorkerPythonThread(self.finishedNotify, "Worker-%s" % i))
    def finishedNotify(self, thread):
    def getThread(self, target, args=(), kwargs={}):
        thread = self.__queue.get()
        thread.setTargetArgs(target, args, kwargs)
        return thread

if __name__=='__main__':

    def realTask(num):

    pool = PoolPythonThread(20)
    for i in xrange(1000000):
        th = pool.getThread(target=realTask, args=(i,))

More information about the pylucene-dev mailing list