[Design] Data migration between Chandler releases

[Sheila Mooney]

So, for a while now there has been an item on the Preview roadmap to  
handle data migration. This means that when users install new  
versions of Chandler (ie: 0.7.1 or 0.8) they have the ability to  
migrate all their data to the new version. You may have seen this  
listed on a planning page or in previous discussion described as  
"dump and reload" or "installer work". We haven't talked about it for  
a while so I thought it was worthwhile to give an update on where we  
are.

For the past couple of milestones, we have had some partial solutions  
in place, with the ability to save and restore settings. This  
restores all shared data but users still have to manually export and  
re-import any non-shared calendar data. Currently people use a  
combination of techniques ie: stamping all their tasks as events  
exporting then re-importing. We need something more end-to-end for  
Preview that handles all the data types and doesn't require users to  
piece together a bunch of different work-flows. PPD has an  
outstanding task to write up a more formal spec for the end user  
experience we want to have when people install new versions of the  
app. In order to do this, we first wanted to understand some of the  
details around how installers work so we can spec out something  
taking into account any technical limitations.

So Bear, Mimi and I met last week to discuss how installers work on  
all the various platforms. The meeting was fairly brief but complete  
notes of the meeting are available here http://wiki.osafoundation.org/ 
Journal/InstallersDiscussion20070118.

In summary, we came away with the following conclusions/proposal....

+ Since installers work different on all the various platforms, it's  
probably NOT a good strategy to develop sophisticated installers that  
prompt the user through a series of steps in order to migrate their  
data.
	+ Typically Windows installers have much more flexibility. We can  
detect that the user has a previous version installed and prompt them  
to migrate through a series of wizard-like dialogs.
	+ Right now, for the Mac, we simply replace the existing executable.  
We would have to write something custom in order to simulate the  
installer type wizard.
	+ Linux is similar to the Mac - we replace the existing executable.
  	+ We can certainly write custom installer program for the Mac or  
Linux but this is not very customary and considerable work.

+ Instead of using the installers, we came up with the following  
alternative proposal...
	+ We can check for a previous repository and version of Chandler  
when the application starts up.
	+ Today we already do some detection on startup ie: we know there is  
a repository from a previous version of Chandler.
	+ The easiest scenario is to have the user manually dump out the  
data from the old version of Chandler and manually reload the data to  
the new version using menus.
	+ We can prompt the user that they are running a new version and  
they should do this.
	+ Possibly we can look at automating one of these steps but at a  
minimum a manual dump and reload would be good enough for Preview.

+ Some questions/issues we are working through....
	+ There is still an issue when a user installs a new version and  
replaces the old one. They might not have dumped out their data. In  
the worst case, we might have to instruct them to reinstall the  
previous version and then backup their data. There are likely  
alternatives to consider ie: prompt to backup when you quit the app.
	+ Should we also explore the effort required to potentially open up  
a previous version of the repository and dump into the new one?
	+ What happens when you dump the data - ie: pre-defined file name  
and location?
	+ Can we handle settings as well as data ie: accounts, preferences etc?
	+ How often to we anticipate the schema to change after Preview?

As a next action, Mimi has started putting together some work-flows  
which describe the user experience....

************************************************************************ 
***************************************
In Preview, we would like to consolidate the following functionality  
into a single workflow:
+ Restore shares
+ Save and restore settings
+ Backup and Restore

MANUAL Workflow
1. Go to File menu and select 'Back-up data'
2. Chandler translates user's data into EIM and saves it as a .txt  
file somewhere it knows about.
3. User installs new version of Chandler
4. User goes to File menu and selects 'Restore data'

AUTO-MAGIC RESTORE Workflow
1. Go to File menu and select 'Back-up data'
2. Chandler translates user's data into EIM and saves it as a .txt  
file somewhere it knows about.
3. User installs new version of Chandler
4. On install, new Chandler detects that this is a version upgrade  
for the user
5. New Chandler finds user's data in EIM format and imports it into  
new version of Chandler

POTENTIAL UPGRADES TO THIS WORKFLOW
1. User is given a choice as to what data they want to restore (which  
collections)
2. The Back-up portion happens automatically before a new version of  
Chandler overwrites the previous version.

************************************************************************ 
*****************************************

Questions and comments welcome.

Sheila