[pylucene-dev] PyLucene in CherryPy

[joe kim]

> One problem I've come across using xmlrpc (with Twisted) in a web
> environment, is that when traffic is high you run out of file handles
> on your server. 100 web server requests means 100 (or more) HTTP
> connections for HTTP/1.0 -- which is by far the easiest to
> implemenent. (I'm not sure Twisted's xmlrpc service supports
> HTTP/1.1!)

My experiences support what you are saying here.  I successfully
implemented a PyLucene search application ontop of twisted web.
Performance is pretty good under normal conditions.  When I stress the
search application using http_load I get pretty good TPS.  When I
stress a simplier page I stop accepting after about 30 seconds, but
after a short interval twisted web will start accepting connections
again.

Maybe twisted web2 can handle this better, but I am a little hesitant
to delve too deep with twisted web and it's divmod incarnations.  On
first glance, things do not appear clean and stable enough for my
purposes.  And this is from a guy who tried to run PyLucene in
CherryPy/TurboGears.

> So let's say you have 4 round-robin web servers all hitting a central
> server. If each of those takes 500 hits, that means 2,000 connections
> to the web server. Over PB, it means 4 connections.

I need to do more research on PB.  Are you saying that PB can function
as a HTTP load balancer?

> And as Pete Fein mentioned, xmlrpc serialization isn't always fast,
> especially for large, complex data structures.

There are always tradeoffs.  A binary protocol can be blazing fast,
but a major pain to develop with.  Personally, I really like JSON.
Straight up data is much easier to model across languages and
serialize over the wire than function calls.

Joe