[Scooby-dev] Re: [Cosmo-dev] Cosmo/Scooby Merge (Please read and comment)

[Brian Moseley]

i've addressed some of these concerns previously in the thread, so
i'll only respond to new ones.

On 7/14/06, Jared Rhine <jared at wordzoo.com> wrote:

> Does this follow as a pro?  Scooby can become "the user view" of Cosmo
> regardless of integration of the code bases and release management, correct?
>
> I hope it's fair to say that Scooby can not lay claim to be *the* user view
> of Cosmo.  Chandler desktop already provides another Cosmo view and will
> continue to do so.  I have personal command-line scripts which provide
> "views" on Cosmo data.  Beta and 1.0 timelines state formal client support
> for only Scooby, Chandler, and the Hosted Service interfaces (3 views to
> Cosmo right there), but no doubt we all hope and dream of "user agents"
> numbering in the dozens.

i think you're pouncing on an imprecision in john's language and
turning it into something tht he didn't mean. his point, as i
understand it, is that scooby and cosmo's web uis would merge into the
most commonly used (but by no means the only possible) dashboard for
interacting with cosmo data.

as i've said numerous times in the thread, and with which katie and
others have agreed, offering as many possible ways for clients to
speak to cosmo is one of our core principles.

> Elsewhere you said that it didn't make sense to issue CalDAV to localhost.
> I submit that it does make sense.  Whatever the protocol used by Scooby to
> provide its web user interface, it should provide a *level playing field* to
> other client authors.  If we had established the non-feasibility of a
> regular-ole CalDAV requests going over localhost, and found a need to
> supplement performance, I will still argue that any such interface should be
> natively network-enabled and openly published.

there is still a level playing field - caldav and atom. the merge will
not change this fact.

> Direct in-process API calls between a client and a server at this point
> prioritizes short-term performance and beta goals too far over so many
> long-term goals without, I argue, really needing to.  I honestly worry that
> inevitably, there will be things that Scooby can do that no other client can
> do.  Only with unlimited resources could you guarantee that alternative and
> duplicate access methods (such as CalDAV) for every Scooby operation are
> available to others.

part of the problem here is, i think, the misconception that scooby is
some sort of big beefy thing that needs to be separately scalable from
cosmo. far from it. if you look at the current scooby server-side
(java) code, 70-80% of it exists solely in order to talk to cosmo. the
rest exists to transform in-memory representations of calendars and
events into a form that the scooby web client (the javascript running
in the browser) can use.

let's compare the cosmo and scooby internal designs:

cosmo
-----------
interface layer:  dav, atom, web console
service layer: very small amount of business logic
data layer: puts stuff into storage and takes it back out

scooby
-----------
interface layer: javascript, runs in browser
service layer: converts server data into the form used by browser and back
data layer: puts stuff into cosmo and takes it back out

with the merge, scooby's data layer goes away completely, and its
service layer is merged into cosmo's. the amount of scooby java code
is minimal compared to the total amount of cosmo java code. the real
meat of the remaining scooby code is in javascript, and that executes
in the browser anyway, on a completely different tier.

merging the two means that we actually remove a tier that has to be
separately scaled without adding a significant new scalability burden
to the final product.

> If the product merge includes URL/namespace functional collpase/merging in
> order to "solve" the sharing URL problem, I am currently set against it.
> Please, we need a rich, well-defined namespace and semantics, not magic
> do-everything URLs.  Could you perhaps expound on how the ecosystem URL
> benefits from the merge; I'd like to hear more.  Anyone want to take a crack
> at documenting the planned namespace?

the merged ui would either use the current /console or choose a
different name. no other urls will be affected. one of my design
principles is that there are separate url spaces for each interface
into the server. different interfaces ) have different security
policies, and it's much easier to apply those policies to distinct url
spaces.

> I do not think Cosmo should have any web UI at all.  To me, and a great many
> people, Cosmo is our data store, our persistence layer, our backend, our
> daemon.  For 20 years, I've expected servers to be servers and clients to be
> clients.  I don't merge my mail client and my mail server to achieve tighter
> integration and performance.  I don't merge my web server with my browser.
> My database with my desktop or web app.  It's not that no one has thought of
> it or tried, but that successful network-based architectures use the network
> abstraction to provide many benefits that everyone cares about.

there are plenty of server products in the world with built-in web
user interfaces. whether or not you choose to use them is a matter of
taste :)

your skepticism here again seems to be based on the assumption that
merging scooby's web ui into cosmo will somehow affect the
availability of network interfaces to cosmo, and that's just not true.
and it's not true that scooby's web ui will somehow cause cosmo to be
less scalable or performant. the vast majority of scooby processing
time is spent in your browser, not in the server, and the work that
the server does do is equivalent to the work it does to handle a
caldav or atom request.

> I've noticed a trend in the use of the word "ecosystem".  As it seems to me,
> the original introduction of the term to our community was to reflect the
> full extent of all available integration points.  People outside OSAF.
> Outside developers.  External web integrators.  That Web 2.0 mashup stuff.
>
> More and more, I've seen the term confined to the products we are
> developing.  I want to use the word "platform" instead for that scope of
> products (I see Mikeal using the same).

i don't disagree.

> By using direct API calls, the line drawn between Cosmo and Scooby dims or
> greys and that line does not become a channel by which anything else can
> interact.  The most successful ecosystem participants succeed by doing a few
> things very well, not by having an ongoing plethora of
> not-quite-rich-or-generalized-enough interaction modes.

the line originally drawn between cosmo and scooby was arbitrary, imo.
part of the server was chopped out of cosmo and moved into scooby.

all we're doing now is moving the line back to where (i think) it
should have originally been - the line between server and the *real*
client: the browser.

> Ah, CalDAV.  It is indeed a vexing issue.  I'm very pleased for the wide
> support that people have shown for CalDAV, in this and any future
> implementation.  I think there's some logical inconsistency to spending much
> (any?) time on CalDAV support if none of our products consume that protocol.
>  Let's address that in other threads.

i don't. interoperability with other products is still a major goal
for cosmo. even if an organization chooses cosmo as its calendar
server, there's certainly no guarantee that every member of the
organization will choose to adopt chandler or cosmo as their ui. we
don't want to leave those folks out of ordinary everyday calendaring
capabilities, even if they don't get some of the ultrafancy
chandler+cosmo features. and to the extent that we can, we should
attempt to standardize some of those ultrafancy features and convince
other product teams to support them (collections of arbitrary mixed
content is the big one for me).

> Could we get specific on this point as I have concerns that by merging URLs,
> we will lose architectural flexibility in a way that we don't have to.  I do
> support moving any web UI URLs out of cosmo's namespace.

that won't happen. only the web ui's urls will be affected, and there
will still be a single entry point for those.

> I readily admit this would be a boon to development.  Is there some
> architectural work on authentication that could helpfully be done at this
> time?  Perhaps Cosmo and Scooby, as separate applications, can be using the
> same interface to a unified authentication proxy layer?

that's not really necessary at this point. we don't have any ecosystem
requirements (that i know of) to support anything other than tickets
and basic authentication, but all the mechanisms are in place for
people to plug in support for authenticating against external
services. these plugins could be used in either the current
two-project setup or in the merged application.

> > - Merge the Cosmo and Sooby mailing lists.
>
> Let's talk long term, say 2 years.  I submit that in this future, the
> release cycles and concerns for Cosmo-the-backend and
> Scooby-the-all-singing-all-dancing AJAX++ application are going to want very
> different release cycles.  Cosmo will be the "server"; there you want
> stability, probity, deliberation.  Clients, you want to try different nifty
> stuff.  Release cycles are likely to want to be different (as they are now).

i relate to and to some extent agree with this point, but i think the
proof is in the pudding. the only thing we can do to make you trust us
is to prove that we can maintain stability while adding features.

i would urge you to not smirk at this statement while recalling
cosmo's release history to date :) we haven't made a lot of effort to
guarantee any sort of stability or compatibility so far cos, well,
we're a pre-alpha product. when figure out what feature-complete
means, and then get to that point, we can start making guarantees. not
until then.

> A merge appears to hinder external development.  The merged project because
> larger, harder to grok from the outside, harder to decompose and so forth.
> Others have expressed the same.  This shouldn't be a major consideration
> until Beta though.

the merged project is only larger in comparison to either of the
original two. the whole will actually become less than the sum of the
two parts, in terms of size and complexity.

> I firmly believe that collapsing clients and servers, and their communities,
> rates of development, riding concerns, and so forth has significant
> long-term detriments that compromise the long-term sustainability of the
> architecture.  I reject the notion that they layers can be properly
> separated *later* because of the proven difficulty of putting asunder that
> which was once together, at least in software.

i don't think we'll have any need to separate anything.

> I'd like to address in separate emails additional concerns I have about the
> the performance, load distribution, load balancing, session management, and
> related issues.  I don't dispute that "scooby is lightweight" argument, but
> disagree that this means it doesn't matter where it runs.

i'll wait until i hear your specific concerns before i comment.

> Overall, it's appears that I'm most uncomfortable about the
> direct-to-Cosmo-API aspect of the Cosmo/Scooby merge proposal.  Load
> distribution is right behind it; I need scaling capability for the Hosted
> Service, not raw performance.

scalability is one of my top concerns, and i'm convinced that this is
not a detrimental step in that direction.

> Thanks.  I appreciate you and others soliciting and responding to feedback.

thanks for taking the time to share your concerns. this is a big
decision and needs to be well considered.