other formats (Re: [Dev] 4Suite RDF and ZOD)

[David McCusker]

Eric Gerlach wrote: [ snip ]
> Making alternative storage methods possible needs to be something
> that happens within the Model of the MVC pattern.  To allow for
> multiple storage technologies, one has to make another layer (which I
> would recommend, if only so that the storage layer could be used in
> other apps.)

Yeah, that makes sense. While models don't depend on views in MVC,
the views do depend on models, so it's up to a model not to commit to
a single storage technology making views interact with a storage API.

Well, if the storage API exposed to views has some abstraction, then
different implementations of the storage can be used.

> A lovely ASCII art rendition of my thoughts (may look wrong if you
> don't use a mono-spaced font):
> 
>        +---->User----+
>        |             |
>        |             v
>      View       Controller
>        ^             |
>        |             |
> +------|--+-----+----|--------+
> |      |  |Model|    |        |
> |      |  +-----+    |        |
> |      |             v        |
> |     Manipulation Layer      |
> |        ^         |          |
> |        |         v          |
> |       Storage Layer         |
> |        ^         |          |
> +--------|---------|----------+
>          |         |
>          +- Data <-+

That's a nice diagram.  I'm generally in favor of more layers, when
each layer acts to protect code on both sides of the boundary from
depending on each other overly much.  The performance impact of extra
layers is generally neglible as long as there are not many layers, and
as long as copying doesn't happen heavily at each layer boundary.

> In this diagram, as per a normal MVC-based approach, the user is
> presented a view (a form), and sends input into a controller (where
> the changes go after pressing the OK button).  However, I've broken
> down the Model into a Manipulation Layer, which makes sense of the
> data, and a Storage Layer, which handles the grunt work of storing
> and retrieving data.

I like the Storage Layer idea.  Basically I'd like a spec for what the
model needs to store, expressed as an interface a model could use for
reading and writing whatever it likes to store persistently.

The least demands are put on a Storage Layer when the model asks for
exactly what it wants to happen as an effect, and leaves it up to the
storage system to make this happen.  But an API constructed like that
can accidentally lock the model into not changing it's mind much.
It's friendlier to incremental design if the storage layer is generic.

If the storage layer has a systematic way of presenting persistence to
the model layer, then a model can recombine the primitives of storage
to meet new demands in the future on the fly, without requiring an
overhaul of a top-down spec for storage based on initial model usage.

> There are two benefits to this design.  One, as I mentioned above,
> the Storage Layer can be separated into its own library and used in
> other applications -- this would be a huge boost to the Open Source
> world.  Second, now the Storage Layer is easily replacable if a
> faster/more secure/more reliable technology comes along, so long as
> it supports the same semantics -- this is what allows multiple
> storage technologies.  (There was a suggestion of e4Graph some time
> ago on the design list, for example)

Having more re-usable libraries for the open source world is cool.
And the prospect of being able to easily replace the storage is quite
useful, both for storage providers and for Chandler folks in testing.
(Sometimes it's easier to isolate causes of bugs by replacing parts of
a system with supposed equivalents to see whether bugs persist.)

If someone can get e4Graph working as Chandler's storage, or any other
fully general storage mechanism, then that sounds like the design for
persistence is on the right path.  I'd guess having both a Python and a
C level API for storage replacement would make sense.  (I need to start
learning about Python internals for C and C++ primitive wrappers.)

> So what really has to be done is to determine the format of data that
> Chandler needs to store while not constraining it to particular
> technologies and implementations.  Once that data model is reasonably
> well defined, RDF and ZODB should be re-evaluated to see if they
> still meet the requirements.  I see no reason they wouldn't, they're
> both great technologies... but you never know.

Oh, I'm sure they'd still meet the requirements based on satisfying the
API and desired semantics.  They might not have the same performance
or same something else as alternatives.

--David McCusker