Support for more detailed report in pipeline output

[konjp]

Currently, when some of test are failing then only useful information that is printed out is "#[error]Failures found in test results" even debug is enabled on pipeline. You need to go to Tests tab to find errors description. This behavior is OK, for default option, but it would be nice with some input task parameter(e.g. 'detailInfo') to control how much detailed is displayed in pipeline output. This is especially nice when new people are added to project, and it is not immediately known that you need to go to Tests tab to find out what are errors about. If you only click on failing task in pipeline you will not see to much info :-).

[konjp]

I think it is ok with this option to enable CLI output, but I think it still missing information about error. Failures found in test results is not enough informative. I know that you go in test tab to found details, but quick info in console should be also nice to see.

Something like this would be nice: -DeploymentTemplate Must Not Contain Hardcoded Uri - azuredeploy.json -Parameters Must Be Referenced - azuredeploy.json

[sam-cogan]

The current implementation tells you which files had tests fail, and how many tests failed for that file. The "Failures found in test results" is the message you get if you don't have cli output turned on, so you may want to check your settings. In my view this provides a good level of detail to help you hone in on where your failed tests are, but then use the test tab to dig into it.

The list of failed tests already exists in the XML we output that is consumed by the test reader, seems a waste to re-hash this again on the CLI, making it harder to read, this is what the test results step is designed for.

[konjp]

Cli output option is turned on, because I also get this in output: ##[warning] '***************\ azuredeploy.json.json failed 1 test Plus "Failures found in test results" message.

I' just saying below every file that contains error can also written just title of test case that failed. And if somebody does not like it, it can be turned on/off with cliOutputResults option . If you run arm-ttk directly(from scripts), without using this task, all information is in console. Therefore, I do not think it is waste to re-hash some of information into console :-).

[sam-cogan]

I'm still failing to see the value in this, the test results task will present all this information for you, in a much better format, why would you not use that? What benefit do you get from using it at the CLI?

[konjp]

It is correct that you have all details in test tab, and in better format. But when you are in development mode, when you have a lot of iterations, deployments, sometimes saving some clicks it matters :-). It is not big issue, just personal preference, and I think this task is way better then using scripts in pipeline.

[Splaxi]

A better error message would make it easier from a PR perspective in Azure DevOps, as the error is "pulled" into the overview:

So the benefit would be that you got "enough" details served directly in the PR overview and didn't have to navigate to the run / test overview.