[pylucene-dev] Fwd: RE: GC Warning: Header allocation failed: Dropping block.

Yura Smolsky info at altervision.biz
Wed Feb 16 14:13:21 PST 2005


Hello, Andi.

>> In general, for a large Java heap, you are better off running
>> a recent gcj.  And ideally the collector should be built with
>> -DLARGE_CONFIG, which I don't think it is by default.

AV> I imagine you are running a recent gcj already since you solved the 4gb file
AV> limit, aren't you ?

yes. I build latest gcj (4.0 now) from cvs.

AV> To build a gcj with LARGE_CONFIG, see PyLucene/INSTALL.

Well, I already built gcj with this option.

Also, I have attached file which can reproduce bug about
OutOfMemory on bix index. I use Queue() to initiate search in the
PythonThread. Maybe Queue() can affect on things?..

Tomorrow I am going to port my example to java.

Yura Smolsky.


_______________________________________________
pylucene-dev mailing list
pylucene-dev at osafoundation.org
http://lists.osafoundation.org/mailman/listinfo/pylucene-dev
-------------- next part --------------
#!/usr/bin/python2.4

from PyLucene import *
from Queue import Queue, Empty, Full
import time

class LuceneSearcher:
    def __init__(self, path):
	self.searcher = False
	self.path = path
	self._reloadIndex()
	self.analyzer = StandardAnalyzer()
	self.queryParser = QueryParser("content", self.analyzer)
        self.sort = Sort()
        #self.sort.setSort([SortField("modified", SortField.STRING, True)]) 
	
    """ just store for all variables of searcher """
    def _reloadIndex(self):
	if self.searcher:
	    self.searcher.close()
	self.directory = FSDirectory.getDirectory(self.path, False)
	self.searcher = IndexSearcher(self.directory)

class LuceneWorkerThread(PythonThread):
    def __init__(self, luceneSearcher, qIn, qOut, id):
	PythonThread.__init__(self, name='LuceneWorker-%s' % id)
	self.luceneSearcher = luceneSearcher
	self.queueIn = qIn
	self.queueOut = qOut

    def search(self, query):
	query = self.luceneSearcher.queryParser.parseQuery(query)
	topFieldDocs = self.luceneSearcher.searcher.search(query, None, 10, self.luceneSearcher.sort)
	for fDoc in topFieldDocs.scoreDocs:
	    fDoc = fDoc.toFieldDoc()
	    print fDoc.doc, fDoc.score, fDoc.fields
	return topFieldDocs

    def run(self):
	while True:
	    #print self.id,
	    method, args = self.queueIn.get()
	    if method=='search':
		result = self.search(args)
	    self.queueOut.put(result)


# main
luceneSearcher = LuceneSearcher('../index/index03')

# for big indexes numberThreads > 5..10 will cause OutOfMemory
numberThreads = 10 
# 

queueWorkers = Queue()
for i in range(numberThreads):
    luceneWorker = LuceneWorkerThread(luceneSearcher, Queue(), Queue(), i)
    luceneWorker.setDaemon(1)
    luceneWorker.start()
    queueWorkers.put(luceneWorker)
	
lw = queueWorkers.get()
# add trigger item to initiate search process
lw.queueIn.put(('search', 'good'))
#wait for response
print lw.queueOut.get()


More information about the pylucene-dev mailing list