[Cosmo-dev] Deployment of Cosmo persistence load distribution

[Jared Rhine]

Brian Moseley wrote:
> you don't think it's true that we don't have a concrete feature list
> for 0.5, or for beta?

No, I just think it's very possible to derive useful feature lists from what
we already know.  I also have to assume that we're making key
architectural/project decisions based on what we know to be coming up.

>> + No major architectural layers get added during Beta; all the layers
>> necessary to scale to large volumes is in place before Beta.
> 
> i'll agree with this only if we have concrete requirements for
> performance and number of supported users.

While there will be targets, and sometimes higher levels of scale do require
rearchitecture, in practice I've found that within very wide ranges, your
deployment either allows you to add servers to distribute load or it doesn't.

For this particular architecture we're outlining, I think we'll be able to
scale up during Beta and later to something like 50 app servers and 20
databases.

I won't be able to state definitely that it'll scale to that size, but if I
can show 85%+ incremental-scaling up to 8 app servers and 3 databases, I
could estimate with reasonable confidence what we could ultimately scale to,
and go live during Alpha with just a couple servers.

> we need to set some realistic goals for launch and then plan how to get
> to higher numbers after.

I'm hoping to advocate a lightweight architecture (as we've done) that can
scale to much larger than what we're likely to see at Alpha or Beta launch.
 I'm asking that we do important work to support that scaling before Alpha
(such as a database lookup service, clustered caches, and distributed lock
support) so that we're not trying to bolt on a major new infrastructure
component frequently to stay just-ahead of the total number of signed-up
users.  Scaling is not *the* most important thing to get done by Alpha
(Beta, maybe), but if it's hard, it should be done early because other
components build on top of it and if it's easy, then we should be able to
find time to do it.

I'm hoping this approach is good news for dev, pretty much, because once the
base load-balancing features we've discussed do get implemented, I think dev
will not have to think about this area for quite a while.  If it's ready by
Alpha, then Beta can be about polishing and getting real-world experience
instead of adding new infrastructure.

>> + 2 MySQL database servers
> 
> master/slave?

No, db1, db2 to show going to a 2nd partitioned database for some portion of
the overall traffic.  Master/slave or active/hot-standby should be added
after load-balancing (but before Beta) because it can be added without any
developer integration (and it won't significantly affect our availability
during Alpha).

> the limiting factor is development resources.

Certainly; the need to understand just how much work is needed to reach
certain levels of service/features is what drives this whole conversation.
If there are enough bare-minimum features to exceed available resources,
that's good to know now.  Balancing features with resources is a key program
management function, but all that work relies on the teams pulling out
high-level lists of work required (for load-balancing, for instance, the
list like distributed caches, locator service, session clustering, etc).

> we're absolutely going to have to get started very soon on identifying
> the features needed for the production launch and planning how to get
> them done.

I suggest that we're already into this process; that's what this thread is
to me for the "infrastructure/deployment" features; the list of
things-we-basically-agree-on about future components has grown nicely as a
result.  Other threads are pulling out other launch features.

> we can possibly start moving forward on other things in parallel. my
> main concern is knowing exactly what needs to be done, and figuring
> out the dependencies between items, so we know what to do first.

While it's difficult to get to "exact" lists much before weeks in advance of
work, I think that conversations like these are helping to get to "pretty
good" lists and some idea of major dependencies with plenty of lead time to
do something useful with the info.

-- Jared