Passing information between Test Steps

[mimir-d]

Issue by dimkup Thursday Aug 27, 2020 at 15:47 GMT Originally opened as https://github.com/facebookincubator/contest/issues/141

---

Intro

There are some cases when it would be useful to pass some information between Test Steps in the same Job/Run.

For example some step could acquire some resource (for example create a block list rule). The resource should be released by some further Test Step, so in order to do it effectively, the first step should pass resource ID to the second step.

Another example: some Test Step produces an artefact (builds a package) and another Test Step should deploy or run the artefact. It would be good to have a way to pass the artefact ID/url to the next step.

The Idea

Currently there is no way Test Steps could pass information to each other. In order to introduce it, we could try to extend the Target structure with kind of context, but there are some issues with the approach:

• The context is "per target", but the information we want to pass between steps in not necessarily "per target". Some Test Step could collect all the targets first and acquire some resource which is shared among the targets.
• The context is shared among Test Steps, so some Test step could damage the context in unexpected way
• The context has to be unstructured. Since we don't know in advance, what kind of information we need to pass, we have to make the context dynamic structure, what is error prone.

There is another approach to the information passing - reuse existing event infrastructure. Generally the only thing we need to do here is to extend Test Step's Run method signature with Fetcher parameter and change existing limitations on event queries. This way Test Steps would be able to emit and consume events from each other. How could it help?

• Event can have any context - "pre Target", "per Run", "per set of Targets", "per Job" etc
• Event is a standalone immutable structure. There is no way it can be changed after the emission.
• Events are specific and could be tied to a specific structure. (It would require additional changes)

Since most of the time Test Steps need to interact inside the same Run, as suggested by @insomniacslk, it's pretty easy to keep them in a local memory cache and heat up the cache from the DB in case of restart.

The issues with the approach:

• There is no warranty that Test Step consumes existing event. (Probably we could force teststep plugins to register as event consumers and check producer/consumer match on start)
• Missing/duplicated events. It would be hard to enforce produce/consume logic in a way each instance of produced event of certain type should be consumed specific number of times. But may be it's an overkill :)
• Cross plugin dependency. Probably we could restrict producer/consumer code to be in one teststep plugin. Of course it would not work for all scenarios, but could be a tradeoff.
• TBC :)