[pylucene-dev] status of pylucene & mod_python

[Ofer Nave]

> -----Original Message-----
> From: pylucene-dev-bounces at osafoundation.org 
> [mailto:pylucene-dev-bounces at osafoundation.org] On Behalf Of Pete
> Sent: Monday, March 26, 2007 7:15 AM
> 
> On Monday March 26 2007 4:21 am, Ofer Nave wrote:
> > > -----Original Message-----
> > > From: pylucene-dev-bounces at osafoundation.org
> > > [mailto:pylucene-dev-bounces at osafoundation.org] On Behalf Of Ofer 
> > > Nave
> > > Sent: Sunday, March 25, 2007 4:22 PM
> > >
> > > [snip] ...the lesson for me here is I need to rearchitect 
> my app so 
> > > PyLucene code never has to be in the apache process.  :(
> >
> > And so I have.
> >
> > I just finished coding up a module I named ipc.py.  It provides a 
> > function called call_in_seperate_process that takes a 
> module name, a 
> > function name, and wildcard args.  It then uses popen2.Popen3 to 
> > invoke the ipc module as an executable, and yaml to serialize the 
> > request and the args.  The ipc module-as-executable 
> deserializes the 
> > request, executes it, uses yaml to serialize the output of the 
> > function, and prints it to STDOUT, where it is then 
> deserialized by the parent process and returned.
> 
> Congratulations! You've just re-invented CGI.

Not quite.  :)  I created an elegant wrapper that allows one to call a
python function from python (with just slightly uglier syntax), but have
that function execute in a separate process.  Very niche, I know, but I'm
still proud of having generalized the solution for my particular problem.
The only thing in common with CGI is that it is client/server with the
client executing the server for each request.  But if models were all that
mattered, then you can say CGI just reinvented inetd.

> Not literally, but the whole create-a-process-per-request is 
> pretty close.  
> You're going to pay the overhead of starting a Python 
> interpreter for each hit (*not* small) and get none of 
> (Py)Lucene's caching benefits (which are a large part of the speed).

Doesn't matter.  The webapp I'm running under apache is an internal use only
app - tools for my team to aid in the Lucene development process.

> So the first page of my requested wiki will be:
> 
> RunningPyLuceneUnderApache:
> 
> DON'T DO THAT. IT DOESN'T WORK. PLEASE STOP TRYING.
> 
> Anyone who disagrees is welcome to edit. ;-)
> 
> Seriously, this just doesn't work.  There are a number of 
> viable alternatives, including mod_proxy, fastcgi, scgi and 
> mod_wsgi (in devel).  Use Apache to front a standalone python 
> process, don't try to run PyLucene under it directly.

So far I agree with you.  Though I'm not ready to give up trying.

> PS to Ofer: You may want to check out the subprocess and 
> pickle modules instead of popen/yaml.  And maybe 'Dive into 
> Python' or  'Python Essential Reference' as well.

Been dying to use subprocess since I started writing Python almost a month
ago.  Waiting on the admins to have time to upgrade from Python 2.3 on my
dev box.

I actually started with Dive into Python, and when I got through that I
grabbed Python in a Nutshell and Python Cookbook.  All three are excellent
and complement each other.

-ofer