[Design] Re:persistent event message queues

[rys mccusker]

Andrew Francis wrote:
>   I think you are mistaking events with their
> manifestion (messages) and the mechanisms used to 
> tell applications about events. 

I don't confuse them. When I said the events were messages,
that was a definition that pertains to my page.  it was
not intended to be true across space and time.  I don't
even believe in such things, and hate to argue with folks
about global definitions.  In practice,
I do tend to treat manifestations as primary, though.

>   Message queues are more analogous to UNIX System V
> messages than UNIX pipes. Messages are for 
> asynchronous communications. UNIX pipes are for
> synchronous communications and are straightforward 
> to use when one is dealing with a single producer 
> and consumer. 

I was trying to provide an analogy for people with minimal
background in technology that would help them think about this.
I was thinking of Inferno channels actually.

yes, my main interest in messages is for async uses.

>   What I suspect is that you want a higher level
> abstraction like the Observer design pattern (from 
> the book "Design Patterns: Elements of Reusable 
> Object Orient Software"). 

I hate discussing things as design patterns.  But the
one you pick is reasonably close.  However, I will not
referred to the observer design pattern again.  Such terms
seem to obfuscate rather than clarify, and help enshrine
the priesthood in technical terms.

I prefer to describe events, or messages if you will,
as one-way notifications.

I won't lecture you on my past experience with model
view controller, and observers, and notification.

 > Producers inform the
> observer about the messages they generate and do
> not worry about the number of consumers or their
> identity.  

yes, and in addition, a producer should not care whether
a consumer has an error, because a producer should not depend
on a consumer.  So a faulty consumer should not interfere
with a healthy producer.

 > Consumers tell the observer which messages
> interest them. When the appropriate message arrivals,
> the observer notifies the appropriate consumers. 

that's one way to do it.  It's an optimization to
negotiate how much communication traffic to send.

>    Tuple space (which you mention) implementations 
> like Javaspaces have a notion of an observer. I think
> a tuple space would be a better abstraction to use 
> for describing an event manager's behavior.

I don't like tuple spaces much myself, because it tastes
like a manifestation of technology which attempts to
promise an effect across space and time which incurs costs
that go up under scaling and interference, and hide the risk
of failure as the difficulty increases.

personally, I would also not care to educate folks using
Chandler about tuple space abstractions.  and I worry
about how many abstractions are necessary to develop code.
but maybe this is a good model for describing this problem.

we can see if Katie is interested in pursuing triple spaces.

>   There should be no doubt whether queued messages 
> are persistently stored. Enqueuing should be an 
> atomic operation whose result is the message is 
> added to the persistent store.

I agree, there is no doubt.  However, the question must be
asked because this is what motivates telling the answer.

>   Do other consumers wait to access the queue until 
> a processing consumer finishes and removes the 
> message? I would assume one would want to remove 
> messages from a single event queue ASAP. 

I asked folks that question so they could see there was
a problem.  John Anderson said an event should stay in the
queue until it has been handled.

Ideally, I would like to see a mechanism where messages
are atomically moved from an input queue to and other queue
which is a persistent to-be-handled queue. but the overhead
for all this might get high without integration.

 > From what I
> have seen, most reliable messaging systems are based
> on the acknowledging the producer once the consumer
> has finished processing the message. 

I would prefer a system where acknowledgment is optional,
and with desirability indicated by the producer in some
"header" in the event consumed.  Basically, I'd like to
annotates events with met information saying how areas
consumers ought to respond, and where to respond, etc.

But I don't really care to go into details now, this is
a huge tension from the current demo priority tasks.

 > In your scenario,
> I suspect the producer and consumers, not the queue
> manager, are responsible for ensuring reliable 
> messaging. The queue should guarantee that messages
> are added and removed. I would be inclined to add
> operations like peek only if they were absolutely
> necessary. Perhaps it would be prudent to look at 
> how reliable messaging systems work (i.e., Java 
> Messaging System)? Also do you really want 
> reliable messaging for something like mouse events?  

yes, I think producers and consumers should negotiate
themselves over liable messaging.  Specific operations
can be discussed by folks pursuing such mechanisms
if we add them to Chandler.

The comparison to user interface events like mouse
events is intended as explanation and motivation, as
well as arguments that messages and user interface events
are not particularly different in nature.  It's not really
a suggestion about specific uses like mouse events.

Rys