[Chandler-dev] Integrating Egg-based parcels in 0.7alpha2

[Phillip J. Eby]

Egg-based Parcels in Chandler
=============================

This document summarizes how egg-based parcels are to be implemented in 
Chandler 0.7alpha2.  Some portions of this have already been implemented, 
while others are still in progress.  (See the `Implementation Plan`_ 
section below for details.)

For 0.7alpha2, we are only supporting the installation of plugins (and any 
supporting libraries) as eggs.  The Chandler core is not being put into 
eggs yet, nor any of the external libraries we depend on, except for 
setuptools itself (which is needed for creating and using other eggs).

During the development of 0.7alpha2, we'll be porting existing plugins like 
the Amazon, Flickr, and Feeds parcels to egg format.  Mostly, this just 
means a slight reorganization of the directory structure, and the addition 
of "setup.py" files describing each plugin.

The proposed directory layout would look like this::

     chandler/
         projects/
             Chandler-AmazonPlugin/
                 setup.py
                 amazon/
                     ...
             Chandler-FlickrPlugin/
                 setup.py
                 flickr/
                     ...
             ... etc.

The new 'projects' directory would be the home for any egg-based project 
that is bundled with the Chandler distribution.

Each subdirectory under 'projects' is a modular, independently 
distributable component that can be packaged in a standard .egg file for 
distribution.  The only things added to the existing code of these 
projects, is the parent directory and the setup.py.

The naming of the project directories is based on an emerging convention in 
the Python community for naming plugin projects registered with the Cheese 
Shop (Python's CPAN).  All Python eggs projects must have unique names 
(case-insensitively), so using a "Chandler-" prefix and "Plugin" suffix 
avoids confusion with any other "Amazon" or "Flickr" packages distributed 
in the Cheese Shop.

This means that we will be able to register plugins with the Python Cheese 
Shop, and users will be able to install them by typing simple commands like::

     release/easy_install Chandler-AmazonPlugin

which will automatically find and download the latest egg from the Cheese 
Shop and install it for use by Chandler.  At startup, Chandler will notice 
the presence of the egg and load any parcels that were registered in that 
project's setup.py.


Creating Plugin Eggs
--------------------

To create a plugin egg, one simply creates a directory with a 'setup.py' 
script, looking something like this::

     from setuptools import setup

     setup(
         name = "MyPlugin",
         version = "0.1",
         description = "This is just an example of a plugin project",
         test_suite = "mypackage.test_module",
         packages = ["mypackage"],
         include_package_data = True,
         entry_points = {
             "chandler.parcels": ["My Plugin Name = mypackage.plugin"]
         }
     )

(There are other fields you can fill out besides these, like license, 
author name, etc., but they are all optional.)

In the directory with this script, you place any Python modules or packages 
that are part of the plugin.  You must make sure that all packages and 
modules are listed in the setup() arguments, and that any modules which are 
Chandler parcels are named in the entry_points argument.

In all other respects, the modules and packages are normal Chandler parcels 
or Python code.  However, you do not need to put the directory on the 
PARCELPATH or PYTHONPATH.  Instead, you run the setup script with either 
"install" or "develop" as a command.  For example, running this::

     RunPython setup.py develop

will add a (temporary) link to your package from site-packages, so that 
your code is automatically importable when Chandler runs.  And, the 
"entry_points" information in setup.py is also registered, so that Chandler 
will autodetect and load the parcels you listed there.

If you want to distribute your plugin, you can do:

     RunPython setup.py bdist_egg

which will create a file like "MyPlugin-0.1-py2.4.egg" in a "dist/" 
subdirectory of your project.  This file, if placed in a directory on the 
PARCELPATH of a working Chandler installation, will automatically be 
scanned for parcels to load when Chandler starts.

Finally, if you want to run your plugin's tests, you can use:

     RunPython setup.py test

assuming you've defined an appropriate test_suite in the setup() 
arguments.  Currently, a test suite must be the name of a module containing 
unittest.TestCase subclasses, or the name of a zero-argument function that 
returns a unittest.TestSuite.

There are many other things you can do with your setup script, but rather 
than detail them all here, I'll refer you to the setuptools developer's guide:

     http://peak.telecommunity.com/DevCenter/setuptools

It covers everything from how to define project version numbers, to 
autotagging SVN revisions, to registering your project with PyPI (aka the 
Cheese Shop), and defining dependencies that get automatically installed 
along with your project.


Build System Integration
------------------------

Initially, the chandler/Makefile will include a BUNDLED_PLUGINS variable, 
listing the .egg file for each plugin.  As part of the normal "make 
install" process, it will download the .egg files and install them.

Initially, this process will be done via curl, but as soon as is practical 
we will switch to using easy_install to do this directly, because managing 
an explicit .egg list will quickly become annoying.  For example, running 
"make install" with a .egg list will wipe out any "setup.py develop" steps 
that a developer has done to work on a plugin.  There are also a few other 
creature comforts that easy_install will give us for plugin development 
that the curl-based downloads cannot.

So, pretty much as soon as we have the basic integration set up and 
working, we'll move to using easy_install instead of curl to download and 
install anything that's packaged as an .egg file.

For full builds, the externals/Makefile will implement the "builds" and 
"binaries" targets for plugins by running their "setup.py build" and 
"setup.py bdist_egg" commands, thus creating the .egg files needed by the 
chandler/Makefile's "make install" step.


Modifying Plugins
-----------------

Because the normal "make install" installs a .egg file with the contents of 
a plugin, there will be an extra step needed to do development on a plugin.

No, you will not need to run "make install" after every change.  Instead, 
you will run "RunPython setup.py develop" in your project *once*.  This 
will switch the plugin to "development mode", where any changes you make 
will take immediate effect.  The plugin will remain in development mode 
until/unless you install an .egg for it (e.g. via running "make install", 
or by using "setup.py install" to explicitly install it as a .egg file).

This step is needed because in the new model of parcel development, there 
is no requirement that a parcel be in any particular directory.  A user can 
develop parcels *anywhere* on their system, not just on PARCELPATH.  So, 
the parcel has to be activated by installing it, either as a .egg file or 
as a "development" link.  The "develop" command installs a special link 
file that tells Python (and Chandler) where the plugin's source code is.

(All of this means, by the way, that it won't actually be necessary to 
check out the projects/ subtree onto a machine unless you're going to be 
changing those plugins, just like you don't have to check out internals/ or 
externals/ unless you're editing that code.)


Test Integration
----------------

The do_tests.sh script will be updated to also run the "setup.py test" of 
each project in the projects/ directory.  If you want to run just one 
project's tests, or a specific test in that project, you can use "RunPython 
setup.py test" in that directory.  The "test" command accepts options to 
tell it what test or suite to run; you can use "RunPython setup.py test 
--help" to get more info, or consult the setuptools documentation.


Implementation Plan
-------------------

Currently, the Chandler startup code has been modified to detect eggs in 
PARCELPATH directories and automatically load any parcels registered in 
them.  (Including "development mode" eggs.)

The next step is to create a "Chandler-HelloWorldPlugin" project, which 
will have no real functionality but serve simply as an example of a plugin 
project.  Its purpose will be to support Bear in adding the new 
build-and-test pieces, before we begin porting any of the current plugins.

Once the hello-world plugin is integrated with the build, we can begin 
porting the existing plugins, which will mainly consist of moving them to 
subdirectories and adding setup.py files.  It would be best if this were 
done by volunteers so that as many people as possible get a chance to 
experience tinkering with a setup.py.  However, it's a simple enough 
process that we could probably do a "porting party" as an IRC office hour, 
with me there to answer questions and take people through a test drive of 
the basic setup.py commands and installing their parcels.