vstest
vstest copied to clipboard
microsoft
Re-Run failed tests using vstest command line options.
I am trying to look for command line switches to pass to VSTest.console command to re-run the flaky failed tests. We have lot of tests that rely on real-time data and often times the tests will time out and fail. When re-run, they will most likely pass. I saw this article and was wondering if this can achieved via vstest command run from command line.
cc @mayankbansal018 can you please direct my question to right team/person.
@pMaske, the feature you have mentioned in not part of TestPlatform, but rather than taht of Azure devops test pipeline.
@cltshivash , @PBoraMSFT , can you please see if it makes sense in platform?
I think it will be a very good enhancement. I think , if relevant tags are provided by platform, this could be easily be achieved by passing values to parameters in test settings file.
Moving this feature into vstest would also be helpful to me because then we could add more parameters and tweak its behaviour. At the moment, because the code is in AzureDevOps, I cannot do that. I'd like to see options on what to do if a test had to be re-run, for example I'd like the test suite to be marked as orange instead of green to indicate that tests were ok but still need to be looked into at some point
I think this would be really helpful. Right now I created my own piece of software only to call the vstest.console and receive the results. If it was the first failure, convert the output to: failed-'NameOfMethod' if it was the second time report the output to TeamCity. I really appreciate if you handle this.
This would be a really helpful feature. We use testplatform in combination with cakebuild. And we have for example one project with 33k Unittests, where sometimes 1-2 Unittests fail through random Timing Problems. It is frustrating, that this flaky Unittests broke often release builds.
This would be a very heplful feature also for me. @Sanan07 Do you have an estimated time for the feature to be released?
@nirasher1 I can't say for now, we still need discussion with the team, if it makes sense to implement on our side. If so, I assume, not earlier that the beginning of the next year.
The Pruner CLI makes this possible.
https://github.com/pruner/cli
But I would like for the Pruner CLI to be able to run tests in a specific order via dotnet test, so that it can run failing tests first, in addition to the already affected tests.
