[WIP] Add Application plugin and Event repeater

[YujiOshima]

This PR add concept of application plugin. ( Related #435 ) Infrakit 'app' is an application that can talk to the infrakit plugins using the plugin api but itself listens on a port and offers a REST api for operations. An example of App is Event repeater. It is subscribe event plugin and publish the messages to other protocol brocker(e.g. mqtt).

** Get Start

*** Prepare Start event plugin and mqtt broker

$ ./build/infrakit-event-time
$ docker run -it --rm -p 1883:1883 eclipse-mosquitto

*** Run event repeater

$ ./build/infrakit-application-repeater -h
Event Repeater Application plugin

Usage:
  ./build/infrakit-application-repeater [flags]

Flags:
      --allowall              Allow all event from source and repeat the event to sink as same topic name. default: false
      --log int               Logging level. 0 is least verbose. Max is 5 (default 4)
      --name string           Application name to advertise for discovery (default "app-event-repeater")
      --sink string           Event sink address. default: localhost:1883 (default "localhost:1883")
      --sinkprotocol string   Event sink protocol. Now only mqtt and stderr is implemented. (default "mqtt")
      --source string         Event sourve address. (default "event-plugin")

$ ./build/infrakit-application-repeater --source ~/.infrakit/plugins/event-time --sink tcp://localhost:1883

Now your app connected to event plugin and mqtt broker. If you set —-allowall, your app subscribe ‘.’ Topic from event and publish all events to broker with original topic. You can specify repeat topics with infrakit command like below.

$ ./build/infrakit application update -h


___  ________   ________ ________  ________  ___  __    ___  _________
|\  \|\   ___  \|\  _____\\   __  \|\   __  \|\  \|\  \ |\  \|\___   ___\
\ \  \ \  \\ \  \ \  \__/\ \  \|\  \ \  \|\  \ \  \/  /|\ \  \|___ \  \_|
 \ \  \ \  \\ \  \ \   __\\ \   _  _\ \   __  \ \   ___  \ \  \   \ \  \
  \ \  \ \  \\ \  \ \  \_| \ \  \\  \\ \  \ \  \ \  \\ \  \ \  \   \ \  \
   \ \__\ \__\\ \__\ \__\   \ \__\\ _\\ \__\ \__\ \__\\ \__\ \__\   \ \__\
    \|__|\|__| \|__|\|__|    \|__|\|__|\|__|\|__|\|__| \|__|\|__|    \|__|


Update application's resouce

Usage:
  ./build/infrakit application update [flags]

Flags:
      --op int            update operation 1: Add, 2: Delete, 3: Update, 4: Read(default) (default 3)
      --resource string   target resource
      --value string      update value

Global Flags:
  -H, --host stringSlice        host list. Default is local sockets
      --httptest.serve string   if non-empty, httptest.NewServer serves on this address and blocks
      --log int                 log level (default 4)
      --log-caller              include caller function (default true)
      --log-format string       log format: logfmt|term|json (default "term")
      --log-stack               include caller stack
      --log-stdout              log to stdout
      --name string             Name of plugin
$ ./build/infrakit application update --name app-event-repeater --op 1 --resource event --value '[{"sourcetopic":"timer/sec/1","sinktopic":"/time/1s"},{"sourcetopic":"timer/msec/500","sinktopic":"/time/500m"}]'

Events are described json style. Then you can delete registerd event.

./build/infrakit application update --name app-event-repeater --op 2 --resource event --value '[{"sourcetopic":"timer/sec/1”}]’

[YujiOshima]

@chungers I want to show you my image with a working code. Of cource I will add tests and documents!

[codecov[bot]]

Codecov Report

Merging #474 into master will increase coverage by 0.78%. The diff coverage is n/a.

@@            Coverage Diff             @@
##           master     #474      +/-   ##
==========================================
+ Coverage   56.56%   57.35%   +0.78%     
==========================================
  Files          57       57              
  Lines        3928     3782     -146     
==========================================
- Hits         2222     2169      -53     
+ Misses       1420     1334      -86     
+ Partials      286      279       -7

Impacted Files	Coverage Δ
pkg/template/template.go	70.55% <0%> (-2.18%)	:arrow_down:
pkg/cli/remote/remote.go	60% <0%> (-0.98%)	:arrow_down:
examples/instance/terraform/plugin.go	61.83% <0%> (-0.54%)	:arrow_down:
pkg/rpc/mux/reverse_proxy.go	39.43% <0%> (+0.97%)	:arrow_up:
pkg/template/funcs.go	64.41% <0%> (+3.09%)	:arrow_up:
pkg/plugin/instance/libvirt/instance.go	58.82% <0%> (+10.15%)	:arrow_up:

---

Continue to review full report at Codecov.

Legend - Click here to learn more Δ = absolute <relative> (impact), ø = not affected, ? = missing data Powered by Codecov. Last update 2218fc7...91c6fe8. Read the comment docs.

[chungers]

@YujiOshima - nice work!

Can you add some docs on the use of MQTT? Also let's add some tests...

There's a infrakit util subcommand that is used to start some useful daemons like the mux proxy (for remote client) and a fileserver (for serving config files/templates during development). Maybe we should just add a subcommand under util for this? Like infrakit util event-repeater and infrakit util event-repeater update?

[chungers]

@YujiOshima - thank you for taking the lead on this.

Let's leave this PR / branch open and refine it. I think you're onto something very cool. My initial thinking when we talked about this was not very refined, but seeing your POC has gotten me thinking about a general API. Can you have a look at my questions and lets discuss more. Thank you!

[YujiOshima]

@chungers Thankyou for your comments! I'm willing to discuss and refine it. First, We need util command and general rest API (of course tests and docs :) )

[YujiOshima]

Hi @chungers ! I added tests and moved commands to under util. And I've made the application publish the REST API instead of rpc. You can use both of unix socket and tcp port. I also modified the commands and APIs associated with it. I updated documents: https://github.com/YujiOshima/infrakit/blob/749e1f2a285fba712105c5b643446442b51668c5/pkg/application/eventrepeater/README.md. Please take a look. Thanks!

[chungers]

@YujiOshima - can you sync and update this PR? Sorry the cmd/cli directory has been renamed to cmd/infrakit plus a few fixes. I will take a look at your README. Thank you!

[YujiOshima]

@chungers Thanks! This PR is now up to date !

[chungers]

@YujiOshima

I get the event repeater but I am little confused by the usage of the infrakit util application tool. I understand how it is used to add / change/ update topics in the event repeater, but how do you envision this work with the rest of the plugin system? In particular, you have this path that can map to resource/resourceID -- do these relate to the plugins?

If the infrakit util application is more closely coupled to the event repeater, then maybe we can structure the commands to be infrakit util event-repeater run and infrakit util event-repeater manage ...?

[YujiOshima]

@chungers Thank you for review. I understand you said and I agree that is more reasonable about event-repeater. But in that case, we will not have any schema for the application. I think it's ok now, but we need discuss about the schema when other application will add.

[YujiOshima]

@chungers I integrated two commands to event-repater command. I remained path option for I want to enable to manage with path. For example, `infrakit util manage post --path /events/EVENTNAME/ --value ... What do you think?

[GordonTheTurtle]

Please sign your commits following these rules: https://github.com/moby/moby/blob/master/CONTRIBUTING.md#sign-your-work The easiest way to do this is to amend the last commit:

$ git clone -b "evrepeater" [email protected]:YujiOshima/infrakit.git somewhere
$ cd somewhere
$ git rebase -i HEAD~842354568776
editor opens
change each 'pick' to 'edit'
save the file and quit
$ git commit --amend -s --no-edit
$ git rebase --continue # and repeat the amend for each commit
$ git push -f

Amending updates the existing PR. You DO NOT need to open a new one.

[chungers]

Hi @YujiOshima - Let's discuss this PR. I think now that we have some useful patterns in terms of controllers vs. plugins, we should revisit this.

I am thinking of the recent example of an enrollment controller that tracks group membership with an instance plugin (e.g. for NFS volume authorization -- see #678 ) could have a nice application here. We can have a controller that maintains a list of topics to register. When the user updates the specification either by add/change/and delete entries, we can synchronize this in MQTT via this controller -- perhaps via calls to "Provision" or "Destroy" an MQTT topic... This would have the nice property of allowing us declaratively specify the MQTT topics from a cluster -- instead of a lot of command line calls that only bash_history remembers... What do you think? Let's talk in person if you are at the Moby Summit in Copenhagen.