[pylucene-dev] status of pylucene & mod_python

[Pete]

On Monday March 26 2007 4:21 am, Ofer Nave wrote:
> > -----Original Message-----
> > From: pylucene-dev-bounces at osafoundation.org
> > [mailto:pylucene-dev-bounces at osafoundation.org] On Behalf Of Ofer Nave
> > Sent: Sunday, March 25, 2007 4:22 PM
> >
> > [snip] ...the lesson for me here is I need
> > to rearchitect my app so PyLucene code never has to be in the
> > apache process.  :(
>
> And so I have.
>
> I just finished coding up a module I named ipc.py.  It provides a function
> called call_in_seperate_process that takes a module name, a function name,
> and wildcard args.  It then uses popen2.Popen3 to invoke the ipc module as
> an executable, and yaml to serialize the request and the args.  The ipc
> module-as-executable deserializes the request, executes it, uses yaml to
> serialize the output of the function, and prints it to STDOUT, where it is
> then deserialized by the parent process and returned.

Congratulations! You've just re-invented CGI.

Not literally, but the whole create-a-process-per-request is pretty close.  
You're going to pay the overhead of starting a Python interpreter for each 
hit (*not* small) and get none of (Py)Lucene's caching benefits (which are a 
large part of the speed).

So the first page of my requested wiki will be:

RunningPyLuceneUnderApache:

DON'T DO THAT. IT DOESN'T WORK. PLEASE STOP TRYING.

Anyone who disagrees is welcome to edit. ;-)

Seriously, this just doesn't work.  There are a number of viable alternatives, 
including mod_proxy, fastcgi, scgi and mod_wsgi (in devel).  Use Apache to 
front a standalone python process, don't try to run PyLucene under it 
directly.

--Pete

PS to Ofer: You may want to check out the subprocess and pickle modules 
instead of popen/yaml.  And maybe 'Dive into Python' or  'Python Essential 
Reference' as well.

-- 
Peter Fein   ||   773-575-0694   ||   pfein at pobox.com
http://www.pobox.com/~pfein/   ||   PGP: 0xCCF6AE6B
irc: pfein at freenode.net   ||   jabber: peter.fein at gmail.com