[Chandler-dev] Automating parcel egg updates

[Phillip J. Eby]

At 02:16 PM 4/18/2006 -0700, Heikki Toivonen wrote:
>So what I propose is that the make
>install automatically builds the egg, then installs the egg.

Note that that would make it impractical to be working on a new-but 
unreleased version of a project on the trunk, at least if we're using the 
current "fixed version number in the Makefile" scheme.  I don't know how 
important that feature is, but from previous discussions here I assumed 
that developers would find it valuable.

>Now this leaves an open question: do we even need to upload/download the
>eggs at all? If the answer is yes, we could make Tinderbox upload the
>newly built egg automatically, and a normal make install would first
>check if there was an egg to download before building one. I'd want to
>avoid the case where the svn repository would need to do an automated
>checkin to update the egg version number in the Makefile, but it would
>be possible. Overall I think the download case would make the system
>much more complex (but doable).

I have an alternative proposal, which will not require any of these 
complexities, or even the existing complexity.

We should simply change chandler/Makefile to specify version ranges to 
easy_install, and allow easy_install to make the decision to either build, 
download, or install, as appropriate.  We already want to do this anyway to 
make other things simpler.

Here is how it would work.  The BUNDLED_PLUGINS would change to be a series 
of setuptools version specifiers, i.e.:

    BUNDLED_PLUGINS="'SomePackage>=1.0' 'OtherPackage>=2.7' ..."

And the installation targets would simply pass this list of requests to a 
single invocation of easy_install, which will validate that they are 
present and installed locally.  If they are not, it will try to download 
eggs from the build servers, or other URLs specified in the Makefile.

This was already proposed as the next step in alpha 3, but I would go one 
step farther, to make it possible for the eggs to be built locally (and 
thus prevent quick builds from breaking when a change is made without eggs 
being available for download).

The change is that we would also pass another option to easy_install from 
the Makefile, or include in a setup.cfg file.  This option would tell 
easy_install what directories the source code of the packages can be found 
in, in the event that a new, unreleased version is required.

Now, the BUNDLED_PLUGINS would look like this:

    BUNDLED_PLUGINS="'SomePackage>=1.0,==dev' 'OtherPackage>=2.7,==dev' ..."

What this says to easy_install is, "I'll also take an in-development 
version of this, if you can find one."  And because 'dev' is a lower 
version number than even version '0', it will only use the dev version if a 
built egg can't be found.

However, once the egg is built from source, it will have a current version 
number, and so will take precedence in future.  If easy_install is run 
again, it will see that the requirement is already met and installed, and 
not bother to search or rebuild anything, until/unless the required version 
is changed in the Makefile.

Now let's say you want to freeze the 'SomePackage' version at 1.0 while you 
work on 1.1.  You would change BUNDLED_PLUGINS to read 'SomePackage==1.0', 
without the '==dev'.  Now you can check in changes to SomePackage without 
affecting anybody else; they will only be able to use existing 1.0 eggs, 
and easy_install will not try to rebuild anything from source.  When you 
are ready to release 1.1, you change the Makefile back to 
'SomePackage>=1.1,==dev' and everyone's next make install will build or 
download.

The only question is whether to bother having checkouts download eggs 
instead of building them, and that can be a configuration 
option.  Specifically, if you don't use --find-links to tell easy_install 
where the source directories are, it won't try to build them, only download 
them.  The tinderbox quick builds would need to be able to build them, but 
individual developers' checkouts don't need to.

At first glance, all of this probably sounds *more* complicated, not 
less.  However, easy_install basically has everything that's needed to 
accomplish the above; the only reason we aren't already using most of what 
I've just described is that Bear requested we make only minimal build 
changes in alpha 2 by keeping the egg download/install process as similar 
as possible to the tarball download/install process.  It was already the 
plan to switch to basically this approach in alpha 3, so the only change 
I'm actually proposing here is to make it possible for easy_install running 
on the tinderbox quick builds to automatically build the eggs when needed.