Add ability to collect scenario data to log it upon failure

[Suremaker]

Description

Usually, scenarios are composed from the steps that somehow alters the service/system under test, i.e. they may be creating/altering some domain specific objects or resources. In most cases, users are not interested in the details such as affected resource Ids, as long as the scenario executes successfully.

The situation is different however when scenario fails. In that case it is very useful to know as much as possible about the scenario context.

So far, LightBDD was not offering any dedicated capability to handle such scenarios and it was required from users to implement it (offering non-perfect workarounds or requiring custom code to be written), including following approaches:

• usage of parameterized steps and putting implementation details parameters in steps to make them reported - this approach causes scenarios to be unclear as it mixes concerns and produces a lot of noise,
• usage of StepExecution.Current.Comment() method to comment steps with additional data - this approach is much better than previous, however produces a lot of noise as comments are added regardless scenario success or failure,
• usage of step decorators and StepExecution.Current.Comment() to include the contextual data comments only upon exception occurrence - this approach is generally the best so far, but requires custom mechanism of collecting the details to be reported upon failure.

This ticket is about introducing LightBDD capability of capturing the scenario data that would be logged only upon scenario failure.

Expected behaviours:

• scenario log context in with internal storage of IDictionary<string, IList<object> type and method to add log entries such as: void AddLogEntry(string key, params string[] values) (append only mode; thread safe)
• scenario log context being easily accessible from any place in code (something like StepExecution.Current.AddLogEntry() or similar),
• the log context being automatically added to the step/scenario upon failure (but not being added on success)
• ability to configure when log context is added (default: upon failure)
• ability to configure how to log context (default: by adding as comment to the step)

Progress

• [ ] Feature is implemented,
• [ ] Ensured backward-compatibility,
• [ ] Ensured good debugging experience

[ipatalas]

I really second that idea! I was actually thinking about writing it on my own - a context which would collect additional debug data which then could be shown only when the test fails (full request/response in case of API calls for instance). Not sure if there is a way to easily intercept when the test fails so to have a place where that context could be dumped. It would be much easier to have it built-in.

[Suremaker]

Thank you @ipatalas. I will bring this feature higher on my to-do list. As soon as I finish the work on the improved reporting, I will take a look at this and will let you know about the progress. Due to the time restrictions, it may take a while however to get it in place.