Eventing Multiple Providers Declarations

[sleipnir]

trafficstars

Although today we only offer support for Google PubSub and logging eventing is on the way with James' work, we were unable to declare different providers for in / out topics. It would be desirable (and expected) that the user would be able to declare different transports for entry and exit.

Using the PingPong project as an example we have the following proto file:

service PingPongService {

  rpc Ping(PongSent) returns (PingSent) {
    option (.cloudstate.eventing) = {
      out: "pings",
    };
  }

  rpc Pong(PingSent) returns (PongSent) {
    option (.cloudstate.eventing) = {
      out: "pongs",
    };
  }

  rpc SeenPong(PongSent) returns (google.protobuf.Empty) {
    option (.cloudstate.eventing) = {
      in: "pongs",
    };
  }

  rpc SeenPing(PingSent) returns (google.protobuf.Empty) {
    option (.cloudstate.eventing) = {
      in: "pings",
    };
  }

  rpc Report(GetReport) returns (PingPongStats) {
    option (google.api.http) = {
          get: "/pingpong/{id}"
    };
  }

I would like to be able to set the in (pings) topic to use a provider such as Google PubSub and the out (pongs) topic to use another supported technology such as MQTT

I believe that by making this separation of transport mechanisms, protocol and or service providers we will be able to cover a wide range of integration options, making our purpose of making Serveless a common place for general programming to get even closer.

What do you think @jroper @viktorklang @pvlugter ?

[jroper]

Yes, I definitely agree that different providers should be supported. The big question for me is at what level should it be configured? The two possibilities I see are either make it a descriptor concern, or deployment concern.

If we make it a descriptor concern, that would mean putting in the gRPC descriptor a provider field so you can select which provider to send/receive this topic from. If we make it a deployment concern, then the gRPC descriptor will just have a topic, but then in the deployment descriptor, we would have a way to map that virtual topic name to a topic on a particular provider, whether that be pubsub, kafka, mqtt, whatever. This configuration might go in the StatefulService descriptor, or perhaps it would be namespace wide and we'd have some sort of Topic descriptor that would map a given topic name to a provider/queue/topic/subscription/whatever.

I think I prefer making it a deployment concern.

[sleipnir]

@jroper I think that definitely the gRPC descriptor should be as clean as possible and therefore we should do this at the deployment level, either in the StatefulService descriptor or via ConfigMap. I would prefer the approach via ConfigMap

[sleipnir]

I like the way we currently declare topics in the gRPC descriptor. I also think that delegating the mapping of the providers / protocols for the deployment is better because it does not imply the recompilation of the application in cases where the protocols / providers of a user function application are changed later even when none of the business logic has been changed and it is just an infrastructure issue