Add event notification

[asadsalman]

I'm writing test reporting and infrastructure around Flank. I want to be notified about various stages of testing. Flank outputs to stdout which is great but it's hard to depend on that for tooling.

Solution: If Flank could either send a request to an end point or push to a message queue when for events like:

1. Shard creation finished
2. APK upload finished
3. Various stages a Test Execution goes through (Preparing device, install app, collecting results, finished etc) Of course this is not an exhaustive list.

It would make it much easier to build tooling around Flank.

Alternative Solution: Use gcloud APIs to get the information in custom tooling myself. This is painful because there's no FTL SDK, I would likely have to use the HTTP API. Flank already has the events I need, when I need them. It just needs to export them somehow, exact exporting mechanism doesn't matter.

[bootstraponline]

Have you seen this?

• https://github.com/firebase/functions-samples/tree/master/quickstarts/test-complete
• https://firebase.google.com/docs/test-lab/extend-with-functions

[jan-goral]

Hi, @asadsalman your feature request sounds interesting. Tell me if those links solve your problem?

* https://github.com/firebase/functions-samples/tree/master/quickstarts/test-complete

* https://firebase.google.com/docs/test-lab/extend-with-functions

If not, IMO we can implement shell hooks in Flank. Flank will call shell commands specified in the config file on the required stages. As it is not obvious to implement, we need to prepare a solution design document, before starting the implementation. If you have any additional requirements or thoughts about the required stages and data that should be passed to hooks, please share them.

[asadsalman]

I did look into functions and they seem pretty useful, but they are are not as granular as the information that Flank shows (examples in OP). I would like to be able to do things like download a shard's data as soon as its available (function's only seem to provide a matrix completion event, not a shard-inside-matrix completion event), measure and track how long different stages take. I think the easiest way to do that is for Flank to notify me of them, since the information's already there in Flank.

[bootstraponline]

we're thinking about making flank a service that would expose APIs.

currently flank is a CLI tool so the integration points are limited.

I think this is potentially a large feature, we should have a clear plan before moving forward.

[asadsalman]

Service based approach sounds like a good idea! Happy to discuss more too, is there a wiki or something used for planning?

[bootstraponline]

We've been writing software design description docs for large features, for example https://github.com/Flank/flank/blob/master/docs/flank-github-action/flank_github_acton_sdd.md

ZenHub is used for planning https://www.zenhub.com/extension

[jan-goral]

we're thinking about making flank a service that would expose APIs.

Right, but implementing hooks will be much easier and faster than making flank a service with exposed API.

Hooks proposal

• The config file will allow specifying a path to the shell command that will be run on a specific flank execution stage.
• During specified stages of execution, Flank will try to run the shell commands delivered in the configuration file.
• Required arguments will be passed by Flank to shell commands.

@asadsalman Let me know if this proposal meets your requirements? If yes, I will convert this issue into an epic and prepare the required tickets for the development. If not, you have to wait for a flank service but currently, we have no epic for it.