[Chandler-dev] Chandler Mail Service Update

[Brian Kirsch]

Hello,
As of revision 14388, the Chandler Mail Service has under gone a 
significant architectural change
 to support downloading of large Inboxes (12,000 plus messages). To 
accomplish this the
Incoming Mail logic has been refactored in to two asynchronous threads 
the Twisted Mail Client thread and the Mail Worker thread. The Twisted 
Mail Client thread handles all downloading and pre-processing of mail 
(building the body text, finding eimml / ics attachments etc). The
Mail Worker thread is responsible for taking the pre-processed mail off 
of a queue and
assigning the values to a MailStamped Item. There are now only two views 
in play.
The Twisted Mail Client view shared by all downloading clients and the
Mail Worker view used to convert pre-processed mail from all Clients. 
With this
check in performance has greatly increased, timeouts have been 
minimized, and
a pesky bus error that occurs when too many Twisted deferreds are on the 
Python
stack has been resolved.


Taken from the r14388 commit message:
======================================
Performance Performance Performance! Complete refactoring of Incoming 
Mail Logic
 to a Mail Worker thread with one view and all clients and Mail Service 
sharing a second view.
The Mail Service can now download 12,000 messages with No timeouts and 
Bus Errors.
There are still performance issues in the UI and Repo layer to tackle 
related to merging,
refresh, and observer calls. But much improvement has been already made. 
Fixed the Bus Error related to too many Twisted deferreds over flowing 
the stack as well as added dynamic
commit calculation and throttling. Improved POP performance 2x when the 
POP3 Server does not support the TOP command (gmail).Added IMAP 
Body.PEEK commands to prevent mail on the IMAP server from being marked 
as seen on download. Added Blocking vs. Non-Blocking logic.
The Incoming Mail Clients can either wait till the Mail Worker has 
processed the last commit request or continue downloading mail while the 
Mail Worker is converting Email Objects to Items.
Added Preview specific Message Parsing and Conversion to Item methods. 
These are intended to maximize RFC2882 text to MailStamped Items 
performance. Improved status bar messages when searching for Chandler 
Headers and when downloading and committing mail. In addition fixed a 
number of open Preview mail bugs.


Please not the Chandler Schema has changed as of rev 14388.

Enjoy,
Brian