[Dev] Twisted has a home on Chandler Wiki

[Dave Smith]

On May 31, 2004, at 7:03 PM, John Anderson wrote:

> I was thinking Macintosh OS 9, which had everything going through the 
> main event loop. Unfortunately, this lead to very clunky multitasking 
> (if you can remember that long ago) and didn't work very well. Things 
> got much better when OS 10 came around and used threads.

I think that the choice between the threads and async model comes down 
to knowing what problem needs to be solved. Clearly both models have 
their shortcomings and are not appropriate in all circumstances. That 
said, it's my personal opinion that the async model is most appropriate 
for systems that need to deal with multiple network I/O sources. While 
the async model does introduce its own set of programming challenges, 
my experience has been that (with the right toolkit) it's easier for 
the average developer to use/build robustly on an asynchronous system.

> I thought threads would actually scale better as we move towards 
> multprocessor machines, but maybe I'm missing something here. 
> Integrating with GUI apps may be a little different in Chandler since 
> most, if not all, the data in Chandler just shows up in the repository 
> as items and the GUI thread gets notified that new data is available 
> to display. I presume the OS support issue is that not all OS's have 
> good thread implementations. This is no longer a problem for Win32, 
> Mac OSX, or Linux -- our target machines. However, this is really a 
> moot point for Python, however, since it runs all it's threads in a 
> single OS thread..

I've had the opportunity to work on some massively scalable network 
servers and the best model that has emerged is a predominantly async 
system with a small thread pool attached. This may very well change in 
the future (+3 years would be my guess) as threading sub-systems in 
O/Ss become able to deal with 100k+ sockets with minimal overhead (i.e. 
no forcible context switching unless a thread already has I/O ready to 
go). Linux has already started down this path with the new PThread 
system.

But as you said, it's moot when you throw in Python with that ever 
lovely GIL. :)

> The Global Interpreter Lock is also cited as a disadvantage of using 
> threads in Python. However, the event model doesn't gain any advantage 
> over threads because of this limitation -- they are on a equal footing 
> -- and if the lock were ever removed, threads would gain an advantage 
> over the event model on multiprocessor machines.

Even if GIL was removed, it's likely that the async model (in this 
particular context) would continue to scale better (where "better" 
means minimizing overall processor utilization) for the immediate 
future.

My $0.02. :)

D.