[Dev] FYI: Schema migration now completed in SVN

[Phillip J. Eby]

Just a quick heads-up: all parcels' schemas are now defined using the 
Python schema API.  I'll be working next on cleaning up the migration tools 
and checking them in so that we'll have them available to help third-party 
parcel developers, but also because we may later want a way to generate 
Python schema from data in the repository (e.g. if we want to create a 
kind->class wizard or something like that, or if we want to encode the 
repository core schema as Python code.)

After that, I'll be working on producing better documentation for the 
schema API.  In the meantime, here's a quick refresher on the main concepts:


Base Classes
------------

The primary schema base classes are Item, Struct, and Enumeration, used as 
follows::


     from application import schema

     class Mood(schema.Enumeration):
         schema.kindInfo(
             displayName = "Mood",
             description = "User's Possible Moods",
         )
         values = "mad", "sad", "glad"


     class SpaceTimeCoordinates(schema.Struct):
         __slots__ = "x", "y", "z", "t"
         schema.kindInfo(
             displayName = "Space/Time Co-ordinates",
             description = "The user's current location in the space/time "
                           "continuum.",
         )


     class MyNewKind(schema.Item):
         schema.kindInfo(
             # kindInfo is optional, and you don't have to set all of these
             displayName = "My New Kind",
             description = "This is just an example",
             displayAttribute = "attributeToUseForDisplay",
         )

         attributeToUseForDisplay = schema.One(schema.String)

         whereYouAre = schema.One(
             SpaceTimeCoordinates,
             initialValue=SpaceTimeCoordinates(0,0,0,0)
         )

         howYouFeel = schema.One(Mood, initialValue="glad")

         whereYouveBeen = schema.Sequence(
             SpaceTimeCoordinates, initialValue=[]
         )

         schema.addClouds(
             default = schema.Cloud(byValue=[whereYouAre,howYouFeel]),
             sharing = schema.Cloud(byCloud=[whereYouveBeen]),
         )


Attribute Definitions
---------------------

There are four kinds of attribute definitions: One, Many, Sequence, and 
Mapping, which correspond to the repository cardinalities of single, set, 
list, and dict, respectively.  The first argument to an attribute 
definition is its type, which may be one of the following things:

* An enumeration class (schema.Enumeration subclass)
* A struct class (schema.Struct subclass)
* A kind class (schema.Item or a subclass thereof)
* A string telling where to import one of the above
* An API-provided shortcut for a core schema type reference (schema.String, 
schema.Integer, etc.)
* An explicit core schema type reference, e.g. 
``schema.TypeReference("//Schema/Core/Kind")``

The current API-provided type shortcuts are Boolean, String, Integer, Long, 
Float, Tuple, List, Set, Class, Dictionary, Anything, Date, Time, DateTime, 
TimeDelta, Lob, Symbol, URL, Complex, UUID, Path, and SingleRef.  These are 
not automatically created, but have to be added to schema.py manually to 
avoid naming conflict with other names exposed by the schema API.  For 
example, ``schema.Cloud`` cannot be used as an attribute type, because it 
is an API for creating cloud definition templates.  To create an attribute 
of type "cloud", you must use 
``schema.TypeReference("//Schema/Core/Cloud")`` as the attribute type.

Strings can be used to refer to types that have not yet been defined.  If 
the type being referenced is in the same module (or is the same class), you 
can use just the class name, e.g.::

      class File(schema.Item):
          container = schema.One("Folder", inverse="contents")

      class Folder(File):
          contents = schema.Many(File, inverse=File.container)

Because ``Folder`` doesn't exist when the ``File.container`` attribute is 
defined, it must use a string to refer ahead, for both the type and the 
inverse attribute setting.  But when ``Folder`` is defined, ``File`` 
already exists, so it can refer to it and the container attribute directly.

If ``Folder`` had been in a different module, however, the type string 
would've needed to name that module, either as a relative name (e.g. 
"Folders.Folder") or absolute (e.g. 
"osaf.somewhere.something.Folders.Folder").  The relative form only works 
for modules that are peers (i.e., that are both directly contained in the 
same package.)

In general, you should use the ``inverse`` keyword rather than 
``otherName`` to set an attribute's inverse, because the schema API can 
perform more error checking that way.  (E.g. it ensures that both sides of 
a bidirectional relationship in fact point to each other).  However, there 
are sometimes unusual schema elements where the reference isn't really 
bidirectional on a type-to-type basis.  That is, there are three or more 
classes participating in the relationship.  In these cases you will 
probably need to use ``otherName``, as you will get schema API errors when 
you try to add a third attribute to a relationship.

The remaining keyword arguments used to define attributes are the same as 
normal repository attribute aspects, e.g. initialValue, defaultValue, 
description, examples, issues, displayName, required, persist, redirectTo, 
etc.  When setting ``initialValue``, you should use the appropriate Python 
expression to create the value, which may not be an exact match of the 
string(s) you would have used in parcel.xml.


Kind Info and Clouds
--------------------

As you've seen in the examples above, the ``schema.kindInfo()`` API lets 
you set attribute values on the Kind, Struct, or Enumeration that the 
schema API generates from your classes.

In addition, the ``schema.addClouds()`` API lets you define your kind's 
clouds in a very compact manner.  It takes a series of keyword arguments, 
each of which defines a cloud alias.  So ``schema.addClouds(foo = expr)`` 
creates a cloud alias ``foo`` that maps to the value of ``expr``.

Ordinarily, the expressions you'll use will be ``schema.Cloud`` 
instances.  ``schema.Cloud`` accepts zero or more "by value" endpoints, and 
keyword arguments mapping inclusion policies to endpoint groups.  So this::

     schema.addClouds(
         sharing = schema.Cloud(emailAddress, byCloud=[contactName])
     )

Creates a cloud named "SharingCloud", with a "byValue" endpoint for 
emailAddress, and a "byCloud" endpoint for contactName.

Typically, you'll be defining endpoints by using the attribute objects you 
just defined for your class, so you don't normally need to put quotes 
around them.  If you're defining something for an inherited attribute, you 
may need to use the superclass name, e.g.::

     schema.addClouds(
         sharing = schema.Cloud(byCloud = [ContentItem.creator])
     )

However, if the attribute is part of the core schema, you may have to use 
quotes::

     schema.addClouds(
         sharing = schema.Cloud( none = ["displayName"] )
     )

Notice, by the way, that using a policy of "none" (in all lowercase) is how 
you can remove a superclass endpoint from a subclass' cloud.

Finally, if you have special needs for an endpoint that can't be met using 
any of these syntaxes, you can list ``schema.Endpoint`` instances in your 
groups, in order to do things like manually set the endpoint's 
``cloudAlias`` or ``method`` attributes.


Parcel Paths
------------

Currently, all kinds have the same repository paths as they did before the 
migration.  The parcel that kinds are placed in is determined by a 
``__parcel__`` constant set at the top of the module.  If ``__parcel__`` is 
not set, the API will consider that module to be a parcel unto itself.  So 
for example, a module ``osaf.framework.blocks.Block`` would be considered 
the parcel ``//parcels/osaf/framework/blocks/Block``, if it didn't have a 
``__parcel__`` setting.  But currently, it has a ``__parcel__`` of 
``osaf.framework.blocks``, so all its contained classes live in the 
``//parcels/osaf/framework/blocks`` parcel.

You need to be aware of this if you care where your kind(s) live, or if you 
need to determine their paths, especially if you are moving them or 
creating new ones.

Another important thing to note is that if a ``__parcel__`` setting is 
used, you *must* add import statements to the named package, to import all 
of the classes defined in the module that used ``__parcel__``.  For 
example, because ``osaf.framework.blocks.Block`` sets ``__parcel__`` to 
``osaf.framework.blocks``, the __init__.py for ``osaf.framework.blocks`` 
has to import every kind class from the ``Block`` module.  This is so that 
the parcel loader can ensure that it knows about the whole schema before it 
tries to load any parcels that depend on ``osaf.framework.blocks``.

So, if you add a new persistent class to a module, please check for a 
``__parcel__`` setting, and if one is present, add or update the import 
statement in the corresponding module or package.

(Note that in the long term, we want to phase out all use of 
``__parcel__``, because it was introduced solely as a way to leave our 
existing repository paths alone and thus minimize disruption while 
migrating to the new API.)


Custom Parcel Classes
---------------------

You can set a parcel's class by setting a __parcel_class__ variable in the 
module that corresponds to that parcel.



Whew.  I think that covers most of the ground.  You can also find more 
detailed documentation on some of these issues in the ``schema_api.txt`` 
file, located in the ``application`` package alongside the ``schema`` module.