[pylucene-dev] Need to build a high-load searcher

[Andi Vajda]

On Mon, 19 Mar 2007, Ofer Nave wrote:

> I need to build a lucene search engine that can handle very high loads
> (hundreds of requests per second) via a web interface.  It will be deployed
> on one or more multi-proc servers, with the index pre-generated and
> available via an NFS partition.  The index is small enough to fit into RAM,
> so assume linux will cache the whole thing (it seems to be now - there's no
> performance difference for me between FSDirectory and RAMDirectory).  I'm
> new to both Python and Lucene, so I have little experience with what the
> best solutions are.
>
> I've looked around for embeddable python web servers, or python http server
> classes, and could only find the ones included in the python standard
> library (like BaseHTTPServer), which frankly suck (though I'm using it for
> development for now, for lack of something better).
>
> My current thoughts:
> 1) As I understand it (and please correct me if I'm wrong), Python threading
> probably won't cut it (especially since I'll likely end up with many python
> extensions to Lucene), so I imagine I'll need a multi-process solution
> 2) I could run under mod_python, though I previously had high hopes of a
> standalone solution for ease of deployment/administration - would be great
> to start the service by just running a python script, rather than
> configuring apache.
>
> Ideas/suggestions/anecdotes?

Usually - see the archives to thislist - the challenge is with using the 
correct thread class, PyLucene.PythonThread, from within the web server 
framework you choose.

Andi..