Event Sourcing support

[icebob]

NATS Streaming, Kafka...etc

[WoLfulus]

We do need reliable messaging in some cases, and controlling the way messages and events will be consumed. For example, I do want some events to be broadcasted to all the services, but some must be processed only for one listening instance, or at least one of a kind (type of service). Because things gets a little problematic while scaling.

Scenario:

• Service "account" that has a "register" action - just register a new user and raises a "user.registered" event
• Service "payment" that creates a client account on the payment gateway (saas) and associates that the new registered account id to the client id

Problems with this:

• I can't have multiple (scaling) payment system nodes listening to the event in order to associate the account (multiple accounts will be created)
• I'd need to write a service (which will run only once - one node) with no actions just to consume that event.
• I have no guarantees that this event will be consumed

I think this scenario also applies to @deividasjackus from what I've seen in gitter

NATS Streaming solves the third problem but not the other ones (?), and I've never worked with kafka so I'll need to take a deeper look into it.

[icebob]

Thanks @WoLfulus! But I think, NATS Streaming can solve all three problem with queue groups, isn't it?

[icebob]

The first point is solved in the next version and it works with any transporter.

[TomKaltz]

Personally I would like to see this event sourcing/streaming model included in Moleculer. Would this be it's own transporter like in #104? Should durable event sourcing streams be accessed from a separate broker used within a service? If it's a separate broker transporter it seems like discovery and health should be turned off for this type of transport.

The other option would be to allow multiple connections within a single broker. One for the standard ephemeral messages and separate connections for durable message stream subscriptions.

@icebob what is your design vision for this feature?

[icebob]

I have no enough knowledge about event sourcing to implement it yet. But I don't think it must be a part of the core modules of Moleculer. Because if you use event sourcing to communicate services, firstly you save the event and send to the consumers. So you don't need to use the built-in Moleculer communication. Thereby it can be service mixin like the queue services, which defines && handles a new queue property in the schema and add a new function to the service instance.

In our case, for example, it creates a sendEvent method which calls publish in event sourcing implementation, and processes the es property in service schema and makes subscriptions on the message broker (Nats Streaming or Kafka). And it can add more replay and snapshot features too to the service.

For example:

module.exports = {
    name: "payment",
    mixins: [EventSourcingMixin()],

    // Event handlers for event sourcing
    es: {
        "order.created"(payload) {
            // Do something useful

            // It calls event sourcing logic instead of send event via transporter
            this.sendEvent("payment.charged", { ... });
        }
    }
}

[zllovesuki]

Currently I get around with this issue by piggy-backing the broker instance with a STAN instance, which connects to the same NATS cluster (well with NATS Streaming somewhere of course). Then I have separate workers (independent of moleculer) to consume and ack on completion.

Not ideal (in fact this seems to couple some things), but an okay workaround.

Though I should really take a look at that queue services mixins. It seems less complicated.

[zllovesuki]

I think the reason why it's so difficult because it's not very intuitive to keep track of the state.

For example, with NATS Streaming, client ID + subscription determines a durable queue. However, with microservices (such as with moleculer), it's not an easy job to store client ID as "stateful", so when these microservices spin up/spin down they can keep track of the what messages should've been deliver to them.

It's easier to have a standalone consumer in which the configuration can be managed outside of moleculer's scope. However, I do see how you can have a publisher integrated with moleculer (such that you can call broker.publish(topic, payload)).

[al66]

I have the problem with loose coupled services also. I have the scenario above described by WoLfulus with registration of members in a group, where dependent on the policy multiple members must agree and I don't want to hard code the possibilities. And also a second scenario with events emit by a stfp watcher for new file arrived, which I may not loose.

My planned solution I'm working on, is a stand alone service "flow" which works with kafka. It has an action for emit events and stores the context (ctx.meta) together with the message (payload as parameter of flow.emit) under the event topic. The same service provides a second action for subscribe - where the parameters are:

• event
• conversion rule for parameters (from the contexts to the action parameters)
• action to call
• conversion rule for the result (how to store the result of the action in the context)

This subscription is stored in a database with the event key and an individual id. For each subsribed event a listener is consuming the topic under the consumer group of the subscription id - therefore the event can be consumed multiple times by multiple subscriptions but only once per subscription id. If a call of the subscriber action falls, the event can be consumed again later on and will stay in the queue of the subscription. If the call of the subscriber action was successful, the listener stores the result in the context and save a new event "step..terminated" in kafka. Further proccess steps can subscribe to this event to consume it and start the following process steps.

If this will work (I'm actually not sure :-)), this will be a general solution for linking actions, which do not know each other.

[icebob]

@al66 it sounds well! If you have some implementation, please share with us.

[al66]

I have started with a prototype - flow.

But I have performance issues with the kafka libaries - actually I mix two of them: kafka-node and kafkajs. Due to performance issues - kafkajs has higher through put for the consumer but about 30% lower speed as producer. But at all the perfomance is low: ~375 rps emit / ~350 rps consume + call action + produce Seems to be a problem of the libaries - I run some checks with native usage (w/o moleculer): kafka-node ~420 rps producing / ~1,17k rps consuming kafkajs ~290 rps producing / ~3,7k rps consuming Calling moleculer actions is only a small overhead - in my environment about 12k rps.

I will go on with it - for my planned solutions it will work. But does anyone knows a better library? Or are these performance issues maybe result that I'm connecting kafka from a windows laptop? (Kafka is running as a docker container on a linux server with ssd and 16GB mem - which should be quite good)

[icebob]

Maybe try the node-rdkafka, It's a native lib wrapper. One disadvantage is that not too easy to use it on Windows.

[al66]

The first version with a static subscription is published - here is the link to npm imicros-flow Not yet well tested, but should work.
@WoLfulus : Should help in your scenario

[TheRaven]

I'm actually really interested in this. We would like to be able use events to decouple service and make sure that if a service is down the event will be consumed later (@WoLfulus' 3rd problem). I was able to build something very similar to @al66's imicros-flow, but right now I have built a series of (really untested and poorly coded) middlewares that allow me to basically override moleculer's event system and send them through NATS-Streaming.

It works without wildcard subscription because nats-streaming doesn't support it yet.

The reason behind it is that I don't really want to have 2 ways to fire events in the system if I can avoid it and I like the way event listeners a registered in a service.

I would say that my main issue though it that i'm not sure doing it this way is a great idea. It feels like It might get hard to maintain whenever there is an update to moleculer.

thoughts?

[icebob]

@TheRaven do you plan to share your solution with public?

[TheRaven]

If I get it to a point where i'm confortable with it yes. I still need to read more about nats-streaming to make sure I can make it useable.

[icebob]

For event sourcing the best option using moleculer-channels