Stream Registry datastore is pluggable

[neoword]

Desired Behavior

Currently the datastore for stream registry is coupled with Apache Kafka and KStreams. For adoption it might be desirable for teams to choose a different datastore.

• Part of this story would be to start with a Pluggable Datastore <Interface> to prepare the existing code to have a pluggable backend.

Benefits

• Hopefully eases adoption
• Enables teams with different backend capabilities to adopt stream-registry

[codeburke]

@neoword Does this include abstracting the name of a Stream from the Kafka topic name? Or is that a larger piece of work that should be tracked as a different Issue (Stream / Implementation abstraction)?

[neoword]

@codeburke, this particular issue is for abstracting out the persistence of stream-registry itself. Right now stream-registry relies on Kafka. There is no reason why it couldn't rely on Postgres, dynamo or Cassandra.

Your question is slightly different. I believe we will need to abstract the stream from the underlying kafka topic name.. certainly while Apache Kafka matures in a 'global context'. This means we need to pick some naming conventions for things.

Maybe something like {namespace}/{cluster}/{streamname}

[codeburke]

Yes, the idea is to provide an abstraction of the name from the implementation details (Kafka Topic name in this case).

What about just {namespace}/{name}? The cluster is already identified in the Stream meta data and that is cross referenced when a Producer or Consumer is registered. That way as a {namespace} you can have a clear abstraction that allows for per cluster configuration by a stream owners without muddying the taxonomy with {cluster} details. This also would also allow for features in the future like cross-cluster failover without breaking the context of “what” stream is actually being talked to in the code. (Ex. I’m publishing to mynamespace/us-east-1/myDomainEvents but due to stream configuration it has failed over to actually producing to eu-west-1.)

[neoword]

All good comentary, @codeburke, lets move that to when we implement namespaces in #29. Your original question was to clarify the scope of this issue, which is purely the SR metastore.