[Dev] Keeping Users' Data

[Morgen Sagen]

On Feb 2, 2006, at 12:25 PM, Phillip J. Eby wrote:

> At some point, Chandler is going to be keeping users' real data,  
> and be the only place where that data is kept.  When that happens,  
> we will be responsible for ensuring that their data remains intact  
> across upgrades.  Even if this is done through some kind of import/ 
> export mechanism, we will need to keep track of the changes in our  
> schema so that this *can* be done.
>
> There are several key questions about this that remain open:
>
> * When will we be committing to keeping users' data safe across  
> upgrades?  Is it 0.7?  0.8?  1.0?
>
> * How will we ensure (procedurally or otherwise) that each version  
> of Chandler will successfully upgrade from older versions?
>
> * What support will we provide for parcel developers to ensure that  
> *their* schemas upgrade safely, as well as the Chandler "out of the  
> box" schemas?
>
> * When an upgrade has to be reverted, what guarantees should we  
> give the user for being able to revert safely without losing data?
>
> While I'm taking responsibility for driving the technical  
> implementation of schema evolution features in 0.7, there is a lot  
> here that is more on the organizational commitment level.  Really,  
> the technical implementation of schema evolution features is only  
> the tip of the iceberg of effort here.  Tracking changes, writing  
> upgrade code, and testing will be the biggest resource investments  
> here.
>
> If you want more technical information about the issues involved at  
> the implementation level, a good place to start is this post I  
> wrote in October:
>
> http://lists.osafoundation.org/pipermail/dev/2005-October/003994.html
>
> You will probably want to skip most of the first half (which deals  
> with developer-only code-upgrading ideas) and go straight to the  
> section entitled "Updating Chandler Schema".  The text that follows  
> addresses details of various aspects of upgrading Chandler schemas,  
> given the existing repository and schema APIs.

I was hoping to have an answer or two, but I only came up with  
questions...  :-)

Although I had always assumed we would have code within each parcel  
that knew how to upgrade its data from the previous version to the  
current version, a more elegant solution would be to have a log of  
schema changes coupled with some transformation code that could apply  
those changes, as you described in your October email.  So how do  
those two methods compare?  You say:

> "With sufficient care and infrastructure support, we can relatively  
> easily
> support manual schema upgrades, in the sense of having installParcel 
> () make
> the changes, if we entirely forbid certain classes of schema change  
> that
> could not be implemented in this way.  However, the amount of  
> developer
> care required currently appears prohibitive, in the sense that it's  
> going
> to seriously impede our flexibility to refactor."

Do you mean that if we did take the route of putting hand-built  
transformation code into installParcel( ), the amount of  
transformation code would be unwieldy?  If we go the schema-change- 
log route, developers would still have to create log entries for each  
change, or are you saying we could automatically build that log?

There's always the chance that if we do the log-based transformation  
system, some parcel writer will want to be able to perform an upgrade  
that the transformation system doesn't support.  In that case could  
they have custom upgrade code in their parcel, or would all upgrading  
need to go through the transformation system?