[Cosmo-dev] Cosmo/Scooby Merge (Please read and comment)

[Jared Rhine]

John Townsend wrote:
> The basic idea is to merge the Scooby Web Calendaring Application into
> Cosmo.

I'd like to review the pros and cons as you present them.

> The pros of this move might be:
> - One integrated project where Scooby's webapp would become the user
> view of the Cosmo server.

Does this follow as a pro?  Scooby can become "the user view" of Cosmo
regardless of integration of the code bases and release management, correct?

I hope it's fair to say that Scooby can not lay claim to be *the* user view
of Cosmo.  Chandler desktop already provides another Cosmo view and will
continue to do so.  I have personal command-line scripts which provide
"views" on Cosmo data.  Beta and 1.0 timelines state formal client support
for only Scooby, Chandler, and the Hosted Service interfaces (3 views to
Cosmo right there), but no doubt we all hope and dream of "user agents"
numbering in the dozens.

> - Possible performance improvements for Scooby (Scooby currently uses
> CalDAV to talk to Cosmo, in an integrated solution, Scooby could be
> refactored to use Cosmo APIs directly)

Yes, this is a truism of system design that you can eek performance
improvements through tighter coupling.  Two points, though: 1) in modern
systems this can be something like 2-5% (or admittedly arbitrarily larger)
for avoiding the network; 2) a successful Hibernate backend will have such a
dramatic effect that any other efforts along these lines smacks of premature
optimization.  May I strike this pro argument as well?  You say "possible"
yourself.

Elsewhere you said that it didn't make sense to issue CalDAV to localhost.
I submit that it does make sense.  Whatever the protocol used by Scooby to
provide its web user interface, it should provide a *level playing field* to
other client authors.  If we had established the non-feasibility of a
regular-ole CalDAV requests going over localhost, and found a need to
supplement performance, I will still argue that any such interface should be
natively network-enabled and openly published.

Direct in-process API calls between a client and a server at this point
prioritizes short-term performance and beta goals too far over so many
long-term goals without, I argue, really needing to.  I honestly worry that
inevitably, there will be things that Scooby can do that no other client can
do.  Only with unlimited resources could you guarantee that alternative and
duplicate access methods (such as CalDAV) for every Scooby operation are
available to others.

> - Better support for the ecosystem. This merge makes solving things like
> the Scooby sharing URL problem easier to resolve.

If the product merge includes URL/namespace functional collpase/merging in
order to "solve" the sharing URL problem, I am currently set against it.
Please, we need a rich, well-defined namespace and semantics, not magic
do-everything URLs.  Could you perhaps expound on how the ecosystem URL
benefits from the merge; I'd like to hear more.  Anyone want to take a crack
at documenting the planned namespace?

> - Common code between the Scooby and Cosmo web facing applications
> allows for better reuse without copying the code between projects. We
> are roll out things like Dojo and they are usable in entire project
> immediately.

Hallelujah!  All for code reuse.  However...

I do not think Cosmo should have any web UI at all.  To me, and a great many
people, Cosmo is our data store, our persistence layer, our backend, our
daemon.  For 20 years, I've expected servers to be servers and clients to be
clients.  I don't merge my mail client and my mail server to achieve tighter
integration and performance.  I don't merge my web server with my browser.
My database with my desktop or web app.  It's not that no one has thought of
it or tried, but that successful network-based architectures use the network
abstraction to provide many benefits that everyone cares about.

> Cons might be:
> - Clients who don't want Scooby would get it anyway (We currently
> deliver both projects as a server bundled release, so clients are
> already use to this. We would want to find a way to allow clients to
> easily configure a scooby-less Cosmo server for non-ecosystem use)

I've noticed a trend in the use of the word "ecosystem".  As it seems to me,
the original introduction of the term to our community was to reflect the
full extent of all available integration points.  People outside OSAF.
Outside developers.  External web integrators.  That Web 2.0 mashup stuff.

More and more, I've seen the term confined to the products we are
developing.  I want to use the word "platform" instead for that scope of
products (I see Mikeal using the same).

We tend to talk about our ecosystem in terms of products, discrete "species"
with the ecosystem.  Ecosystems are far less about the individual objects
within them as they are about the relationships between those objects.  The
members of the ecosystem inter-relate, exchange things, use protocols of
software or nature, and from the hyped "network-effect", an explosion of
exchanges and complexity happen.

By using direct API calls, the line drawn between Cosmo and Scooby dims or
greys and that line does not become a channel by which anything else can
interact.  The most successful ecosystem participants succeed by doing a few
things very well, not by having an ongoing plethora of
not-quite-rich-or-generalized-enough interaction modes.

> - Scooby has been designed to be a Web Calendar that uses CalDAV as its
> data transport mechanism. This serves two purposes: 1. It acts as a
> testbed for Cosmo's CalDAV support and 2. In theory, Scooby could be a
> CalDAV client for another CalDAV server. Turns out that we already have
> Cosmo dependencies that eliminate #2.
> - Less of a need for CalDAV4j. We would still need it for
> interoperability within Scooby (subscribing to external calendars), but
> Scooby would be much less dependent on the library than it is today.
> That could lead to some "bit rot" or certainly less of a focus on this
> component in the future.

Ah, CalDAV.  It is indeed a vexing issue.  I'm very pleased for the wide
support that people have shown for CalDAV, in this and any future
implementation.  I think there's some logical inconsistency to spending much
(any?) time on CalDAV support if none of our products consume that protocol.
 Let's address that in other threads.

> At a very high level, this would mean:
> 
> - Moving the Scooby webapp code (the JSPs, JavaScript, etc.) into the
> cosmo web application.

I'm against this.  Let's go the other way.

> - Move the Cosmo web app (repository browser, admin console) to a
> Spring-based codebase.

Yes, I think the Cosmo web viewing components should be rejiggered to match
Scooby's model, and I propose, integrated into Scooby's codebase.  We should
have only 1 Java web UI framework.

> - Integrate the URL structures of the various webapps so that they made
> sense.

Could we get specific on this point as I have concerns that by merging URLs,
we will lose architectural flexibility in a way that we don't have to.  I do
support moving any web UI URLs out of cosmo's namespace.

> - Make sure that the acegi security implementations are merged.

I readily admit this would be a boon to development.  Is there some
architectural work on authentication that could helpfully be done at this
time?  Perhaps Cosmo and Scooby, as separate applications, can be using the
same interface to a unified authentication proxy layer?

> - Merge the Cosmo and Sooby mailing lists.

Let's talk long term, say 2 years.  I submit that in this future, the
release cycles and concerns for Cosmo-the-backend and
Scooby-the-all-singing-all-dancing AJAX++ application are going to want very
different release cycles.  Cosmo will be the "server"; there you want
stability, probity, deliberation.  Clients, you want to try different nifty
stuff.  Release cycles are likely to want to be different (as they are now).

A merge appears to hinder external development.  The merged project because
larger, harder to grok from the outside, harder to decompose and so forth.
Others have expressed the same.  This shouldn't be a major consideration
until Beta though.

I firmly believe that collapsing clients and servers, and their communities,
rates of development, riding concerns, and so forth has significant
long-term detriments that compromise the long-term sustainability of the
architecture.  I reject the notion that they layers can be properly
separated *later* because of the proven difficulty of putting asunder that
which was once together, at least in software.

I'd like to address in separate emails additional concerns I have about the
the performance, load distribution, load balancing, session management, and
related issues.  I don't dispute that "scooby is lightweight" argument, but
disagree that this means it doesn't matter where it runs.

I do not think the sky is falling.  I find the code reduction, development
speed, management/release overhead, and beta focus issues all compelling
arguments in favor of a merge.  However, I think the practical implications
of the merge will have long-term impacts out of scale with the short-term
gains.  This particular Beta tradeoff appears to hamper post-1.0 targets
unnecessarily.

Overall, it's appears that I'm most uncomfortable about the
direct-to-Cosmo-API aspect of the Cosmo/Scooby merge proposal.  Load
distribution is right behind it; I need scaling capability for the Hosted
Service, not raw performance.

Thanks.  I appreciate you and others soliciting and responding to feedback.

-- Jared