[pylucene-dev] Need to build a high-load searcher

[Ninneman, TJ]

You said it runs perfectly within a shell right?  That is, without the
daemonizing code?  If that's the case, you'll need to scrap that way of
running your app in the background.  

TJ

On Fri, 2007-03-23 at 16:19 -0700, Ofer Nave wrote:
> > -----Original Message-----
> > From: pylucene-dev-bounces at osafoundation.org 
> > [mailto:pylucene-dev-bounces at osafoundation.org] On Behalf Of 
> > Ninneman, TJ
> > Sent: Friday, March 23, 2007 3:57 PM
> > 
> > Are you sharing one QueryParser among many threads?  I 
> > instantiate and descruct a QueryParser within each thread and 
> > have never had any problems.  Like your findings showed, 
> > never call os.fork() within your script or you'll be asking 
> > for trouble (which is why I'm guessing I was not able to use 
> > Flup in pre-fork mode and why you were not able to use that 
> > ActiveState recipe).  
> > 
> > That's why you can use Daemontools to run your app; a fork 
> > system call is never made.
> 
> Might as well just post the source.
> 
> Simplified version of the my daemonized searcher script:
> 
> ---
> # paste
> from paste import httpserver
> # my own libraries
> from mm import daemon
> from mm.search.xmlapi import SearcherApp
> # main
> logfile = open('/whatever.log', 'a+')
> app = SearcherApp(index_dir)
> daemon.daemonize(logfile=logfile)
> httpserver.serve(app, host=gethostname(), port=OPTIONS.port)
> ---
> 
> SearcherApp implements WSGI.  I instantiate one SearcherApp and pass it to
> httpserver.server().  That's it.  I don't explicitly do anything with thrads
> - presumable, httpserver controls thread usage.
> 
> Having a separate SearcherApp instance for each thread might solve the
> problem, but it's unclear from the Paste docs how to do that.
> 
> Also, my testing script is not multithreaded.  It sequentially executes HTTP
> gets against the searcher app.  So in theory, there should only be one Paste
> thread actually do something at a time.  Yet QueryParser still blows up.
> 
> I'm going to try moving the daemonize call before the paste and pylucene
> imports now and see if that helps.
> 
> -ofer
> 
> _______________________________________________
> pylucene-dev mailing list
> pylucene-dev at osafoundation.org
> http://lists.osafoundation.org/mailman/listinfo/pylucene-dev