[Feature Request] Could we introduce a new context for ordered tests?

[StarlightIbuki]

For example:

step("connect to server", function()
  -- ...
end)

step("send requests", function()
  -- ...
end)

The second test should happen after the first no matter what arguments are given, and the second is skipped when the first is failed/skipped.

I know the same behavior can be achieved by putting them into the same it() block, but:

1. It will only be counted as 1 test in the summary;
2. it won't benefit from before_each() etc.

[alerque]

Are tests not run in order now? The flag --sort-tests already defaults to off, and currently we have no parallelization, all all tests are done one at a time in order. Is --no-keep-going what you are looking for? Skipping can be handled by tagging related tests with the same tag to skip.

All this isn't to say we can't consider adding such a feature, but without having threading in the first place it seems like kind of a moot point, no?

[StarlightIbuki]

@alerque The idea is that this context keeps its order even when we set --sort-tests or --shuffle to true, and persists in its behavior when --keep-going is true. After all, we see more clear intentions with explicit statements.

[StarlightIbuki]

And no, it's not the same to use --no-keep-going. I want it to skip its siblings but not other subsequent tests.

[alerque]

Is it possible to achieve your dependency situation by nesting existing blocks?

[StarlightIbuki]

@alerque No given what I read in the doc.

[Tieske]

How would this work if one selects only a single test from a list of steps, using a --tag for example? something to cater for or would that be a user error? (and should it be called out as such?)

[StarlightIbuki]

@Tieske It should be a user error I guess? Or simply make it skipped due to the dependency?

[alerque]

I would imagine the "test" unit that would be skipped / tagged would be whatever is above the steps, not the steps themselves.

[alerque]

I probably won't be able to work on this any time soon, but I would try to facilitate a PR is somebody else did.

[Tieske]

@StarlightIbuki busted has been modeled after some other frameworks, how was this case handled there? That might give some pointers as to how to solve it.

(this case is quite fundamental to how Busted works, hence asking)

[StarlightIbuki]

@StarlightIbuki busted has been modeled after some other frameworks, how was this case handled there? That might give some pointers as to how to solve it.

(this case is quite fundamental to how Busted works, hence asking)

Hmmm. Good point. I did not think much before. I remember some of the frameworks make some assertions grouped as a "test point", which needs to be all passed to count for one success test point, and they are just a part of the test flow. Some of the test frameworks explicitly claim the dependency of a test (like with the notation "@depend.before(test1)"). And for some of the frameworks, this is simply not considered. I guess we have some freedom on how to do it.