Find or document a way to exclude failing scenarios

[tdiesler]

For ./manage run -d acapy -t AIP10 I see ...

Failing scenarios:
  features/0025-didcomm-transports.feature:28  Create DIDExchange connection between two agents with overlapping transports -- @2.1 DIDExchange connection with both agents using WS for inbound and outbound transport
  features/0036-issue-credential.feature:81  Issue a credential with negotiation beginning from a credential request
  features/0036-issue-credential.feature:96  Issue a credential with the Holder beginning with a request and is accepted
  features/0037-present-proof.feature:171  Present Proof where the verifier and prover are connectionless, prover has proposed the presentation of proof, and is acknowledged -- @1.1 
  features/0037-present-proof.feature:172  Present Proof where the verifier and prover are connectionless, prover has proposed the presentation of proof, and is acknowledged -- @1.2 
  features/0037-present-proof.feature:188  Present Proof where the verifier rejects the presentation of the proof -- @1.1 
  features/0160-connection.feature:52  Inviter Sends invitation for one agent second agent tries after connection
  features/0160-connection.feature:69  Inviter Sends invitation for one agent second agent tries during first share phase
  features/0160-connection.feature:82  Inviter Sends invitation for multiple agents
  features/0160-connection.feature:125  Establish a connection between two agents but gets a request not accepted report problem message -- @1.1 
  features/0160-connection.feature:126  Establish a connection between two agents but gets a request not accepted report problem message -- @1.2 

It'd be good to have a way to (temporally) exclude tests with an associated issue, similar to this.

Excluding tests that are known to fail, can help to establish a reliable baseline of stuff that is expected to "always work" and hence make detecting regression easier.

If not already possible, I suggest to find a way to have parameterized exclude sets, which ...

• can have exclude conditions (e.g. -a acapy -b acapy ...)
• exclude the given scenarios (or variants) from the test run
• include what is excluded ;-) in the final report
• link every exclude to an associated issue for details

I'm happy to give this a shot if you like.

[nodlesh]

Is there any other requirement you have that wouldn't be met by using the @wip tag that is already there in the feature files? @wip is standard practice in behave tests to mark Work in Progress tests. Anything not tagged @wip should be expected to pass, at least with one or more aries frameworks.
So your command would have to be ./manage run -d acapy -t @AIP10 -t ~@wip

For tests that are not WIP and have an outstanding issue, then they are tagged with @allure.issue:<issue url> This convention is automatically picked up by the allure reports will provide a link and a small bug icon for the test.

[swcurran]

The problem with @wip is that is global, and in the AATH context the test might only be failing for a given framework/test agent.

The way we have been "managing" (I use the term lightly -- we haven't been too focused on this) this is that in the nightly runs, we have a REPORT_SCOPE for a given "runset" (a term I made up for a given combination of test agents) that should define what is expected to work for that combination of agents. So getting that right is a second way to do this. It would be nice to have a way to easily execute a runset, e.g. ./manage runset acapy-afgo that did whatever was called for in the acapy-afgo GitHub Actions workflow.

I quote "manage", because we set those and don't really change them much over time, even as the frameworks and tests evolve. It would be good to have more focus on tests that fail, to create issues in an appropriate place for fixing the tests -- removing them from the scope as unsupported, fixing the test, fixing the backchannel or fixing the framework. That takes some additional collaboration that we need to generate.

[nodlesh]

Yes very true @swcurran. Even if the @allure:issue: convention is used, it will still get reported as a bug with a link in allure even if it works for the frameworks in that particular runset.

I'm sure you remember us talking about the discovery features protocol where we would query the framework for what it supports and construct the run set on the fly. Not sure if there are any extensive implementations of this protocol, but if not, maybe we can maintain something in AATH that acts like a feature discovery mechanism. Maybe just a file of mappings.