[Dev] Twisted has a home on Chandler Wiki

[Brian Kirsch]

John,
Although Twisted does advocate using an asynchronous model it has very 
good Thread support and Thread pooling
built in to the Reactor core. If we make the decision to leverage 
Twisted as a core element of Chandler it does not
mean we have to give up threads. I created some sample programs that 
leverage the Twisted thread model and they
have performed quite well.

Using Threads:
http://twistedmatrix.com/documents/current/howto/threading

Sample Programs:
http://aloha.osafoundation.org/~bkirsch/src/IMAPTwistedThreadTests.zip




Brian Kirsch - Email Framework Engineer
Open Source Applications Foundation
543 Howard St. 5th Floor 
San Francisco, CA 94105 
(415) 946-3056 

On May 31, 2004, at 6:03 PM, John Anderson wrote:

>
>
> Andi Vajda wrote:
>
>> Summing up my thoughts and understanding about the threading model for
>> Chandler, quite possibly flawed at the moment...
>>
>>  - the UI is in one thread, the main thread. We don't want it to 
>> block or
>>    appear frozen.
>>
>>  - the Twisted model strongly advocates against threaded networking 
>> and
>>    proposes an event loop or upcall programming pattern that 
>> serializes all
>>    the networking operations on one thread. (This reminds me about 
>> network
>>    programming in Windows 3.11, before Microsoft had networking in 
>> Windows).
>>    There are plusses and minusses to this approach, I'm not advocating
>>    for or against it.
>>
> I was thinking Macintosh OS 9, which had everything going through the 
> main event loop. Unfortunately, this lead to very clunky multitasking 
> (if you can remember that long ago) and didn't work very well. Things 
> got much better when OS 10 came around and used threads.
>
> The rational for not using threads in twisted was described int the 
> Intro link as:
>
> "The event-loop networking model was chosen over threads as it tends 
> to be more scalable, integrates well with GUI applications, and is 
> supported by virtually all operating systems."
>
> I thought threads would actually scale better as we move towards 
> multprocessor machines, but maybe I'm missing something here. 
> Integrating with GUI apps may be a little different in Chandler since 
> most, if not all, the data in Chandler just shows up in the repository 
> as items and the GUI thread gets notified that new data is available 
> to display. I presume the OS support issue is that not all OS's have 
> good thread implementations. This is no longer a problem for Win32, 
> Mac OSX, or Linux -- our target machines. However, this is really a 
> moot point for Python, however, since it runs all it's threads in a 
> single OS thread..
>
> The Global Interpreter Lock is also cited as a disadvantage of using 
> threads in Python. However, the event model doesn't gain any advantage 
> over threads because of this limitation -- they are on a equal footing 
> -- and if the lock were ever removed, threads would gain an advantage 
> over the event model on multiprocessor machines.
>
> Multi-threaded programming is tricky, but so is event driven 
> programming -- just go read the twisted explanation on how to use the 
> reactor..
>
> That said, even though I can't see any advantages to the event driven 
> model, this doesn't mean we shouldn't use Twisted.
>
>>  - the Task Manager needs to run a number of tasks in the background,
>>    providing a flexible way of scheduling and prioritizing them.
>>
>>  - the repository transaction model imposes that no two repository 
>> views
>>    share item instances since they are typically of different 
>> versions.
>>
>> With that in mind, my opinion, for what it's worth, is that we should 
>> have the
>> following threading model:
>>
>>  - a UI thread
>>  - a twisted thread
>>  - a task manager thread pool to which tasks are assigned and 
>> scheduled
>>
>> All Twisted-based networking should be done via the twisted thread, 
>> which
>> involves a fair amount of inter-thread communication, but that's what 
>> the
>> Twisted model is all about isn't it ?
>>
>> Andi..
>> _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
>>
>> Open Source Applications Foundation "Dev" mailing list
>> http://lists.osafoundation.org/mailman/listinfo/dev
>>
> _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
>
> Open Source Applications Foundation "Dev" mailing list
> http://lists.osafoundation.org/mailman/listinfo/dev
>