[pylucene-dev] pylucene and 2gb limit of files

Yura Smolsky info at altervisionmedia.com
Thu Aug 10 00:25:05 PDT 2006


Hello, Andi.

AV> On Wed, 9 Aug 2006, Yura Smolsky wrote:

>> check it
>>
>> AV> I believe I've seen this too. Patches are welcome :)
>>

AV> Thanks, Yura, it seems to make sense. On Windows, for 'ren' to work, a file
AV> can't exist under the new name whereas on Unix, mv overwrites the target if it
AV> exists.

yes, thats what python manuals tells.

I am going to implement fully functional python directory same to Java
Lucene one. I will provide it here later, so you can include into PyLucene
if somebody will need.

Also, can I contribute pool of PythonThread classes into PyLucene? This class solves
problem (memory leak) of multiple recreation of PythonThread classes.
I think somebody asked same question here before. I have attached
class.

OMG. I jave just tested it on windows and linux with new PyLucene
2.0.1. I see that we have mem leaks on windows! We have not them
before on 1.4.3 - I am sure. And we have not them on linux on 2.0.1
atm. Andi, what could it be?

Thanks.

--
Yura Smolsky,
http://altervisionmedia.com/
-------------- next part --------------
#!/usr/bin/python2.4

from PyLucene import *
from Queue import *
import time

class _WorkerPythonThread(PythonThread):
    def __init__(self, funFinished, name=None):
        PythonThread.__init__(self, name=name)
        self.__target = False
        self.__args = ()
        self.__kwargs = {}
        self.__funFinished = funFinished
        
    def setTargetArgs(self, target, args=(), kwargs={}):
        self.__target = target
        self.__args = args
        self.__kwargs = kwargs
        
    def run(self):
        try:
            if self.__target:
                self.__target(*self.__args, **self.__kwargs)
            else:
                raise RuntimeError("Target function is wrong")
        finally:
            self.__funFinished(self)
        
class PoolPythonThread:
    def __init__(self, amount=10):
        self.__queue = Queue()
        for i in range(amount):
            self.__queue.put(_WorkerPythonThread(self.finishedNotify, "Worker-%s" % i))
    
    def finishedNotify(self, thread):
        self.__queue.put(thread)
            
    def getThread(self, target, args=(), kwargs={}):
        thread = self.__queue.get()
        thread.setTargetArgs(target, args, kwargs)
        return thread


if __name__=='__main__':

    def realTask(num):
        pass

    pool = PoolPythonThread(20)
    
    for i in xrange(1000000):
        th = pool.getThread(target=realTask, args=(i,))
        th.start()


More information about the pylucene-dev mailing list