[pylucene-dev] poll: threading support ?
aurora00 at gmail.com
Thu Jan 6 20:26:46 PST 2005
I don't have an opinion on the technical approach. My main concern is
on stability. Same thing about Python 2.4 those final version is
barely 1 month old. As long as things work I'll be happy. I guess I
just need some time to test out everything.
As of pre 2.4 Python it is better to say up front that things have to
be done in the main thread only. Putting all Lucene activities in the
main thread in a multithreaded program is not straight forward, but it
On Thu, 6 Jan 2005 16:18:44 -0800 (PST), Andi Vajda
<vajda at osafoundation.org> wrote:
> I'd like to ask how you all feel about the following proposal for threading
> support in PyLucene. As said yesterday, the current approach to threading is
> backwards and very kludgy as it relies on undocumented, unsupported, partially
> implemented inner functions of gcj's boehm-gc component.
> The new approach delegates the creation and initialization of the actual
> OS thread to libgcj which gives it a chance to properly register it with
> boehm-gc, without the use of any kludges such as attachCurrentThread().
> The gcj thread is being passed a python implementation of the java Runnable
> interface, the newly created gcj(java) thread, once started, immediately calls
> back into python grabbing the GIL. This seems to work quite well, the gcj
> thread, apart from being treated as a daemon thread by python, plays well in
> the python environment, no deadlocks, no weird things, several threads run
> more or less concurently (within the usual limits of python's GIL).
> The new approach relies on several things however:
> 1. python's PyGILState_Ensure() and PyGILState_Release() to work properly
> and not deadlock
> 2. gcj's threading implementation to be reliable and not deadlock
> 3. the gcj thread and python thread implementation to be the same
> 1. requires python 2.4
> 2. requires gcj 3.4.2+
> 3. is true on Mac OS X and Linux, where pthreads are used, and Windows
> So, here is the proposal:
> - Support threading with python 2.4 only, threading support code would not
> be compiled into the _PyLucene extension when built against pre-2.4 python
> - Remove the old attachCurrentThread-based threading support altogether. It
> is quite broken. Hans Boehm said he's aware of the issue and might support
> registering threads in such a way at a later time, not in the near future
> though. At this time it only works reliably on Mac OS X. On Windows, the
> threads cannot be unregistered and on Linux, with a 2.6 kernel, the
> process is killed very quickly (bug 2291).
> Resolving the threading issue removes one of the last hurdles to getting to a
> 1.0 release of PyLucene. If there are no major objections to this proposal,
> I'm prepared to release a PyLucene 0.9.5 next week.
> What do you think ?
> pylucene-dev mailing list
> pylucene-dev at osafoundation.org
More information about the pylucene-dev