Dapr demo [do-not-merge]

[wcs1only]

@JochenKienzle and I have been exploring a possible integration with https://github.com/dapr/dapr.

Summary

As part of the exploration, I wrote up a quick and dirty hack to allow dapr to act as a replacement for the protocol gateway layer. This is largely a POC demo so that we could do a basic feasibility test and explore memory usage, ect. I'm creating this PR so that others can try it out, and so that we can open a dialog about how we might build a first class integration, should we decide to go that route.

Running the code

You should be able to run this on your local machine provided you have docker installed. To run, use the following steps (MacOS/Linux):

1. Install dapr on your local machine

wget -q https://raw.githubusercontent.com/dapr/cli/master/install/install.sh -O - | /bin/bash
dapr init

2. Configure dapr to use mqtt

cat > ~/.dapr/components/pubsub.yaml <<EOF 
apiVersion: dapr.io/v1alpha1     
kind: Component
metadata:   
  name: pubsub  
  namespace: iotea
spec:                 
  type: pubsub.mqtt
  version: v1                               
  metadata:
  - name: url
    value: "tcp://127.0.0.1:1883"
  - name: qos
    value: 1
  - name: retain
    value: "false"
  - name: cleanSession
    value: "false"
EOF                      

3. Checkout this branch and start the docker containers:

#Assuming you've checked out this branch in ~/iot-event-analytics/
cd ~/iot-event-analytics/docker-compose
docker-compose -f docker-compose.mosquitto.yml up --build &
docker-compose -f docker-compose.platform.yml   up --build

4. Navigate to your talent directory and execute it with dapr

dapr run --app-id talent1 --app-port $DAPR_APP_PORT node index.js

5. You can publish a test message to the ingestion topic

dapr publish --publish-app-id talent1 --pubsub pubsub --topic "ingestion/events" --data '"{\"feature\":\"anyfeature\",\"type\":\"anytype\",\"subject\":\"test\",\"instance\":\"test\",\"whenMs\":'`date +%s`'999,\"value\":19}"'

Outstanding Questions

1. In this quick-and-dirty implementation I hacked protocolGateway.js directly, but presumably it could live as a separate adapter, or we could do away with protocolGateway entirely. I'd love to hear your thoughts about the best approach is.
2. Dapr needs a signal to say "I'm done setting up my subscriptions, they are safe to be read by the sidecar". I added a start() method to protocolGateway to achieve this, but I realize there can be N protocolGateway instances. Is there a better interface I could be using for this, or should we add one?
3. If this patch were to be made into a state that it could be merged upstream, would it need to be non-breaking and backwards compatible? I'm thinking mostly about the changes I made to the docker container initialization presumably we'd need to flag that, or have a separate set of dockerfile definitions which could be switched between at build time?

Let me know if you have any questions for me. I'm definitely not expecting a line-by-line review, but would be happy to hear any feedback about my initial approach.

[JochenKienzle]

@wcs1only Many thanks for providing this PR 👍

Regarding your questions:

1. ProtocolGateway vs. ProtocolGatewayAdapter The ProtocolGateway is the abstraction layer to provide different adapters. To keep this abstraction and flexibility an implementation of a DaprProtocolAdapter would be the way to go.

2. ProtocolGateway.start() Good point. So, this is a valid extension of the ProtocolGateway abstraction layer and the place where it is called seems also correct. If I understand you correctly then start() implies that all configuration is done an the ProtocolGatewayAdapter can be started. What is open for me is the handling after start() --> What happens if some subscription will be changed? In this case, I assume we have to signal (and restart?) the ProtocolGatewayAdapter, right? I'm not sure if this is an issue, because a Talent (and other components) should define all of its required services/topics/... at start(). @llakie Do you see a need for a dynamic change for platform topics at runtime?

3. Compatibility At the moment, compatibility is needed. In general, I think there is more configurability needed to re-use common parts of the dockerfiles (more modular / layered approach). For now, I think separate dockerfiles are the fastest way to keep compatibility. Improvements of the configuration should be done later. @llakie Do you see any better solution?

Documentation Beside that, it would be cool if you provide a short documentation (markdown) about the adapter and how to start/use it (as you have done it in the first comment of this PR 👍 ) --> e.g. below docs/...

Again, many thanks 👍

[llakie]

Hi @wcs1only, thank you for contributing,

Ad 1: The Protocol Gateway defines a topic based pub/sub interface to all components of the platform no matter, which underlying ProtocolAdapter(s) are used. The dapr implementation should go on the transport-layer, which is a ProtocolAdapter implementation. By using this layer, we can avoid any changes on the platform side and maintain protocol independence.

Ad 2: ProtocolGateway.start(): We do not have any start method as of now, since we assume a lazy initialization in the adapter. As soon as someone subscribes to a certain topic, the adapter has to create the connection, subscribe e.g. to an underlying broker... Therefore a start method should not be necessary, since it's started, if it's used. I assume, that you are doing some kind of configuration, when the subscribe() method is called to know all topics before the actual subscription is done in the start() method. Can you think of dynamic subscriptions at runtime? We also have to consider wildcards within the topics (like # or +). @JochenKienzle So currently, there are no subscription changes on runtime, since everything is resolved via wildcards.

Ad 3: We are always preferring compatibility (in size, performance, dependencies), since we have an unknown number of runtime device types. We should not introduce any changes to the basic Dockerfiles themselves. @JochenKienzle We should have a registry (locally or remote) and derive docker images from there and put the other stuff on top. We could investigate in a Shell-Script, which is executed in an ONBUILD command in the base image to add additional configuration on top.