cli: run a subset of the test suites

[skrrb]

added to anchor test the flag --run PATH --run ANOTHER_PATH ... for running the subset of test suites located under the specified paths

closes #1828

[armaniferrante]

Please add a test for this.

[skrrb]

is there anything else needed from my side to have this merged?

[mralbertchen]

Would it be possible to incorporate this into the next release? I spend probably 30 minutes a day deleting Test.toml and restoring them in order to run certain test suites lol.

[skrrb]

Would it be possible to incorporate this into the next release? I spend probably 30 minutes a day deleting Test.toml and restoring them in order to run certain test suites lol.

in the meantime, you can always clone the repo ([email protected]:skrrb/anchor.git), checkout to cli/run-test-subset, build and use the generated anchor binary

[mralbertchen]

Would it be possible to incorporate this into the next release? I spend probably 30 minutes a day deleting Test.toml and restoring them in order to run certain test suites lol.

in the meantime, you can always clone the repo ([email protected]:skrrb/anchor.git), checkout to cli/run-test-subset, build and use the generated anchor binary

Awesome tip. Thanks!

[vercel[bot]]

@armaniferrante is attempting to deploy a commit to the 200ms Team on Vercel.

A member of the Team first needs to authorize it.

[mralbertchen]

@skrrb do you know why the tests are failing now after the repo update?

[skrrb]

@skrrb do you know why the tests are failing now after the repo update?

seems to be an unrelated clippy warning, already fixed in #2075

[sohrab-]

This is so close! Anything I can do to help?

[mralbertchen]

Any chance for this to make it into 0.26.0?

[skrrb]

hey @armaniferrante! is there anything else missing?

[mralbertchen]

I've created a Docker Image from this PR's branch because we really needed the functionality for CI/CD on our 30-test-suite repo.

If anyone's looking for a solution, you can pull mralbertchen/solana-dev:subset-testing from docker hub. The Dockerfile is open source: https://github.com/mralbertchen/solana-dev-env

[mralbertchen]

Hi @Henry-E looks like you are maintaining the repo now. What do we need to get this merged in?

[Henry-E]

sure i guess the criteria are mainly

• is this a features lots of people want / need or is it a fix for something broken
• are there tests for it
• are all the CI tests still passing

It looks like this has some conflicts which might need to be cleared up but if the tests pass after that we can definitely look into including it in the coming version.

[mralbertchen]

Thanks @Henry-E very helpful. @skrrb are you able to resolve the conflicts and give it one more shot?

Looks like one of the workflow failures is the vercel authorization issue. What do we need to fix that? @Henry-E

Also on the topic of use case - we actually already use this in production because we have 4 programs and around 30+ test suites. We use this to run subsets of tests in CI based on what programs have been changed. I think it's very useful and that's why I have been paying attention to this PR since it was written over half a year ago. It seems like the final missing piece from the test suite release. At the end of the day, is there a purpose (beyond just a fresh validator environment) in separating the test suites if you can't specify a subset to run? This feature has been incredibly helpful to us in the past few months. Want to get it official otherwise I fear our repo will be stuck on this weird version of forked cli in the future.

[Henry-E]

The vercel thing is just a boilerplate error, don't worry about it. We're going to be done v26 pretty soon so I can just look into fixing the merge conflicts.

Just to check though, it's not possible to get this same behavior from just using a custom script under the [scripts] tag in the Anchor.toml file?

Also confirming that the usage for multiple individual tests is repeated `--run x/y --run x/z' commands and that there isn't some alternative list syntax. Probably will add that information to change log since the example test only demonstrates calling a single test.

On Sat, Dec 3, 2022, 7:09 AM Albert Chen @.***> wrote:

Thanks @Henry-E https://github.com/Henry-E very helpful. @skrrb https://github.com/skrrb are you able to resolve the conflicts and give it one more shot?

Looks like one of the workflow failures is the vercel authorization issue. What do we need to fix that? @Henry-E https://github.com/Henry-E

Also on the topic of use case - we actually already use this in production because we have 4 programs and around 30+ test suites. We use this to run subsets of tests in CI based on what programs have been changed. I think it's very useful and that's why I have been paying attention to this PR since it was written over half a year ago.

— Reply to this email directly, view it on GitHub https://github.com/coral-xyz/anchor/pull/1864#issuecomment-1336101103, or unsubscribe https://github.com/notifications/unsubscribe-auth/ADAHMGAFTJOWNMB4FYD3JH3WLLWYRANCNFSM5U446IKA . You are receiving this because you were mentioned.Message ID: @.***>

[mralbertchen]

The vercel thing is just a boilerplate error, don't worry about it. We're going to be done v26 pretty soon so I can just look into fixing the merge conflicts. Just to check though, it's not possible to get this same behavior from just using a custom script under the [scripts] tag in the Anchor.toml file? Also confirming that the usage for multiple individual tests is repeated `--run x/y --run x/z' commands and that there isn't some alternative list syntax. Probably will add that information to change log since the example test only demonstrates calling a single test. …

Ah. I see the confusion. The value of this PR is not the ability to run individual tests. Like you said we’ve had that ability since the beginning with the [scripts]. It’s the ability to run a subset of test suites by specifying the directory. Let’s say I have 5 test suites under the unit-tests folder and 5 test suites under the integration-tests folder. I can pass in —run unit-test to run the 5 tests under unit tests folder and ignore the 5 under integration. This was what was missing after the test suite feature came out in 0.24.0. Anchor natively searches for all the Test.toml to run. But many times during development and CI you just want a subset of that. There was no way to do that until this PR.

[Henry-E]

Perfect, well I will fix the merge conflicts and try to merge

[Henry-E]

multiple_occurences is now the default behaviour in clap (i think) unfortunately though this test doesn't confirm that. If anyone runs into trouble with using the --run command multiple times, e.g. --run X --run Y please let us know.

Assuming the tests pass this time around, it should be good to merge.

[mralbertchen]

@Henry-E I can't thank you enough for this. Thanks so much for being a chad and pushing this through.