Specify a sequence of scenarios to execute in a configuration file.

[raweaver]

Is your feature request related to a problem? Please describe. No, the request is not related to a problem.

Describe the solution you'd like We are using Gauge for automated system testing of embedded systems and we would like to run a specific scenario, to program the Device Under Test, before all of the other scenarios. We currently execute Gauge twice. Once to run the programming scenario then again to run the test suite.

We are currently using tags to group scenarios into test suites and we use tag expressions to run the suites. However, as the suites get large and we have more tests, we thought a configuration file would make it easier to visualize and control which scenarios run in which test suite.

We realize that Gauge appears to execute spec-by-spec. We aren't looking to run scenario 1 from spec A, then scenario 2 from spec B then scenario 3 from spec A. Maybe the configuration file is a YAML file where you specify a sequence of specs and then within each spec you specify which scenarios to execute. For example:

specs:
  - Spec Name A
    scenarios:
      - Scenario Name 1
  - Spec Name B
    scenarios:
      - Scenario Name 2

Then to execute Gauge, you might run:

gauge run specs --sequence specs.yaml

Describe alternatives you've considered We've considered using the BeforeSuite hook for the programming, but still felt that overall it would be nice to specify the complete set of scenarios to execute in a test suite.

We've considered using a script to feed the list of scenarios to Gauge using the --scenario option, but had concerns about the input buffer size on the command line.

Additional context Nothing at this time.

[sriv]

So far, we've been recommending that Gauge specs/scenarios remain independent and this sort of sequencing leads to temporal coupling. And it increases the complexity in executing a single scenario or even rerun all/failed tests.

But this has been requested too many times, so I am going to consider this. (@zabil - thoughts?)

My take is that

• if this were to be implemented, it should not be just a static list (it could be so for a first cut), rather it should be something similar to how build tools work (tasks depend on tasks).

• The file should include the ability to either specify a spec or scenario, or a folder, or even a tag expression.

• There needs to be a check for cyclic dependencies.

• the parallel run needs to be thought over, Does gauge allow users to specify the streams? or does gauge generate parallel streams taking dependencies into consideration? or does gauge just not allow parallel run when it does not control the orchestration?

Alternatively, could we attempt to make gauge honour piping (like unix tools do) so one could do something like cat workflow.txt | gauge run

Thoughts?

/cc @getgauge/core

[zabil]

if this were to be implemented, it should not be just a static list (it could be so for a first cut), rather it should be something similar to how build tools work (tasks depend on tasks).

Preferably a json file. Maybe worth expanding the manifest.json to add something like execution groups roughly something like

groups : [ { name: "sequence1", "specs" : [ { name: "Specification 1", scenarios : []...

gauge run --group "sequence1"

he parallel run needs to be thought over, Does gauge allow users to specify the streams? or does gauge generate parallel streams taking dependencies into consideration? or does gauge just not allow parallel run when it does not control the orchestration?

I assume running in parallel should be disabled as it depends on the order?

Alternatively, could we attempt to make gauge honour piping (like unix tools do) so one could do something like cat workflow.txt | gauge run

If this can also be done on windows/powershell

[sriv]

Did a quick check, and as long as gauge honours stdin, users should be able to pipe in/redirect list of specs/scenarios into gauge.

The challenge though is that gauge does not honour order of scenarios, so this will still be viable only to control the order of execution of specs and not scenarios within the spec.

@raweaver - would this approach work for you? I'll spend some more time once I know that this can be useful.

[raweaver]

@sriv @zabil Thank you for the responses and suggestions.

We discussed the options and we feel that a sequence file passed to gauge as a command line argument would be our preferred solution. Also, controlling the order of specs (not scenarios) would be sufficient. It might be beneficial to structure the sequence file such that you could add support for specifying the scenario sequence in the future if desired.

Thank you!

[samuelbenison]

@sriv @zabil - Any update on this? Would love to see this new feature implemented, this would be really useful for us. Any timeline on when we expect to see this feature released? Thank you so much for adding this new feature.

[irynahryshanovich]

Hi, @zabil @sriv. Do you have any updates about this new feature? It will be very beneficial for our team

[edeleong1493]

Hi all, anyone knows how to run a test case from another test case using gauge?