[Dev] Proposed package layout and API-creation guidelines

[John Anderson]

I'd also like to see a clear division between instances/schema that all 
CPIA applications share (e.g. most blocks definitions), those that are 
specific to Chandler or some other application (e.g. main view trees of 
blocks and the sidebar), and other extensions (e.g. repository viewer 
and flicker)

John

Phillip J. Eby wrote:

> Background/Scope
> ----------------
>
> Last week, Katie, Morgen, Ted and I got together to review the current 
> packaging of Chandler and the progress of the flattening project.  We 
> also talked about creating some patterns for Chandler's API 
> organization.  This post is my attempt at summarizing our discussion 
> and hopefully consensus on how Chandler APIs should be organized, 
> within the following overall scope:
>
> 1. Get API packaging/grouping settled in 0.6.  Given that this will 
> likely involve more flattening and flinging directories around, 
> getting the overall groupings settled is all we can reasonably expect 
> to do in 0.6.
>
> 2. Examine individual APIs more closely in 0.7.  This will be a good 
> time (or at least a better time than now) to revisit Chandler's 
> content model in general, the implementation of stamping, and 
> inheritance vs. composition questions in general.
>
>
> Goals/Issues
> ------------
>
> Given the scope, the goals of the API guidelines are to:
>
> Make Things Easy to Find and Use
>    One of the 0.6 API goals is that someone should be able to create a 
> parcel like the Amazon or Flickr parcels without needing someone from 
> OSAF to sit over their shoulder.  Currently, however, you have to 
> either have fairly intimate knowledge of the codebase or else copy 
> things into your parcel that you don't understand and can't fix.  This 
> is partly because we don't...
>
> Distinguish Internal Components From APIs
>    Right now, there are no organizational cues that distinguish 
> internal components from APIs in the codebase, and in many cases there 
> are no cues to even tell the difference between a module and a class 
> within that module, that can produce confusing errors for people not 
> as familiar with the codebase.  This isn't just a "newbie" problem, 
> either; providing organizational cues is more like making a handle 
> ridged so you can get a better grip on it: it makes things easier for 
> everybody, all the time.  And speaking of making things easier, we 
> also want to...
>
> Provide Easy API Access in Scripting/Testing Tools
>    Our API strategy needs to consider tools like "headless", 
> CPIAScript, and any future embedded-in-Chandler developing or 
> scripting tools.  They need a way to access APIs that's consistent 
> with the way they're used in Python.  If the way we're accessing APIs 
> in Python is "too hard" to be used in these other tools, then the way 
> we're accessing APIs is too hard, period, and we need to make it 
> better, rather than making "headless" or CPIAScript into isolated, 
> training-wheels-only ghettoes.  Similarly, we want to...
>
> Follow Python Community Lessons-Learned
>    ...so that we don't create an isolated island of Chandler-only 
> rules.  We're not in a position to dictate standards, and in any case 
> the common Python practices have solid reasoning behind them.  Minor 
> deviations (like allowing "_" in module names) aren't a big deal, but 
> the overall look-and-feel of the APIs shouldn't produce any big 
> surprises.
>
>
> Proposed Guidelines
> -------------------
>
> Don't Stop Application Work to Refactor
>     Obviously, application feature work takes priority.  It's also 
> best to keep reorganization a separate task from making actual changes 
> or additions, because then it becomes hard to tell what's a problem 
> with a move, versus what's a problem with a change.
>
>
> APIs are short, all-lowercase, top-level names under "osaf"
>     Each API will be accessible via "from osaf import apiname".  For 
> example, "from osaf import pim, mail, startup" would import the pim, 
> mail, and startup APIs.  You can then use expressions like 
> "pim.Contact" to refer to a particular class or function.
>
>     Notice that this means that APIs that are packages *must* import 
> any API classes, functions, etc. from the package's internal modules, 
> to make them available for direct access from the API namespace.  
> Users *must not* need to import modules from inside an API package, 
> and the only time third-level imports may be used is from within the 
> API package itself.  For example, "from osaf.pim.items import 
> ContentItem" should not appear in any code outside the osaf.pim package.
>
>    Instead, "from osaf import pim" and "pim.ContentItem" should be 
> used wherever possible, so that we have a consistent message as to 
> what a thing is called, and the same name can be used in all of our 
> tools.  For example, the Python interpreter for the "headless" tool 
> can import the APIs so that it suffices to refer to 'pim.Contact' to 
> use the Contact type.  The same applies to CPIAScript, etc., so that 
> we have only *one* name by which a given thing is known, across all 
> tools.
>
>    (Since a given API symbol is always used with its API name, we may 
> also want to rename some APIs for a better fit; for example, if there 
> were a MailMessage type exported from the mail API, it might make 
> sense to rename it to mail.Message.)
>
>    In the top level 'osaf.__init__' module, we will have an '__all__' 
> list containing the names of API packages that conform to these API 
> guidelines, so that tools like "headless" can use "from osaf import *" 
> to obtain all of the APIs.  (If an API package does not yet conform to 
> these guidelines, it must not be added to this list.)
>
>
> APIs Are Also Defined Using '__all__'
>    API modules should define an '__all__' list at the top of the 
> module, listing the names of API classes, functions, constants, etc. 
> in that module. (See application.schema and osaf.startup for 
> examples.)  The pydoc tool and "import *" both use this list to 
> control their operation, and it gives people reading the code a quick 
> list of what is actually part of the API.  It frees you from having to 
> use '_private' names for classes or functions you don't want to 
> include in the API, or which are not yet ready for inclusion in the 
> API.  '__all__' is also useful for assembling a package API from 
> individual modules' API; more on this below.
>
>
> Modules are Better Than Packages
>     Don't create a package for an API unless there's too much code.  
> What "too much code" is can vary, but anything over 2500 lines is 
> probably too much, and anything under 1000 probably doesn't make sense 
> to split.  If in doubt, err on the side of keeping it a single module, 
> because someone browsing the code can skim a single module more easily 
> than they can dive into a subdirectory and poke around in more 
> modules.  (Especially if you put an __all__ list at the top of the 
> module so they know what the API is.)
>
>     Keep in mind also that packages are not for categorization.  It's 
> easier for people to scan a flat list of mixed APIs like "app, pim, 
> mail, startup, webserver, sharing, etc." then to delve into a 
> many-leveled tree that perfectly categorizes the code.
>
>     If you're creating an API module under osaf, you can put its tests 
> into osaf.tests, and you can put any doctest files adjacent to the 
> module.  (E.g. 'startups.txt' is adjacent to the 'osaf.startup' 
> module.)  This is a great opportunity to include programmer overview 
> documentation and testable examples, right where they can be found 
> when browsing the source tree.
>
>
> Use Packages When a Module Gets Too Big
>     But *only* if it's really too big.  It's easier to expand a module 
> into a package than to go the other way around.  When you split a 
> module into multiple modules, you should make sure each module define 
> the right '__all__'
> list for its contents, and add a '__parcel__="osaf.whatever"' setting, 
> to ensure that the entire package goes in the same parcel.  The 
> __init__.py for your new package should then use 'from foo import *' 
> for each module within the package, so that existing code still 
> works.  You should *not* give either of the split modules the same 
> name as the old module; this is confusing.  Instead, break up the 
> original module along thematic lines and name the two or more modules 
> for those themes within the overall API scope.
>
>     You can leave your package's tests under osaf.tests if you like; 
> or move them directly to your package directory.  Since by convention 
> we name all test modules beginning with "Test", it is not necessary to 
> create a 'tests' subpackage.  (Because the Test* modules are visibly 
> different and alphabetically grouped, as well as eventually being the 
> only modules that should be using UpperAndLower case names.)  This 
> does not mean that we are flattening 'tests' subpackages; this doesn't 
> have enough API impact to be worth doing.  But it's also not really 
> worth creating new 'tests' packages either.
>
>
> 0.6 Plans
> ---------
>
> The 'pim' package is already in transition towards meeting these 
> guidelines, although there are still many flattening operations 
> pending, such as moving all of the calendar modules up one level, and 
> eliminating the subparcels for contacts, tasks, mail, etc.  These 
> won't take place until after the sets branch is reintegrated and stable.
>
> The plan for the current osaf.pim.mail module is that most of it 
> should be moved to osaf.mail.__init__, except for the five classes 
> needed to support the core pim contentmodel: AccountBase, 
> EmailAddress, MIMEBase, MIMEContainer, and MailMessageMixin.  All of 
> the other classes and APIs make more sense associated with the 'mail' 
> API, as an API for the mail service and mail client operations in 
> general.
>
> For 0.6, the APIs we end up with should be at least pim, mail, 
> sharing, startup, and webserver.  Also, as part of the parcel.xml 
> elimination process, the osaf.examples.* packages are going to become 
> top-level packages under the 'parcels' directory, the way that 'feeds' 
> and 'photos' already have.
>
> _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
>
> Open Source Applications Foundation "Dev" mailing list
> http://lists.osafoundation.org/mailman/listinfo/dev