microsoft
`Microsoft.Testing.Extensions.TrxReport` should allow customizing `trx` file name via dedicated `MSBuild` property
Summary
The only way to currently customize the .trx filename when using MTP with the Microsoft.Testing.Extensions.TrxReport extension is by using the much more general purpose TestingPlatformCommandLineArguments MSBuild property.
I believe forcing the use of this single property makes for a more convoluted configuration experience and creates maintenance/flexibility issues for consumers.
A dedicated, more specific property should be provided that only concerns itself with the format of the trx file and nothing else.
Background and Motivation
We integrated .trx-based test reports in our GitHub workflows this week and it was a pain to get things to how we wanted. The current defaults for the file name are, in my opinion, awful, so we wanted to change them to something that would be more readable to the outside.
We used the dorny/test-reporter GitHub action to parse the .trx files from each of our projects and generate the test report. The tool relies on the filename to produce the various sections in its report (which by itself makes sense).
We made it work by configuring the TestingPlatformCommandLineArguments property like this:
<TestingPlatformCommandLineArguments>--report-trx --report-trx-filename $(MSBuildProjectName).trx</TestingPlatformCommandLineArguments>
Which worked, but with some caveats.
The main issue is that this basically forces us to now always output a trx report, even when we don't want to. Because the way these properties work (as cmdline arguments) it is not possible to only specify --report-trx-filename, otherwise dotnet test will crash with an error saying that --report-trx-filename must be used in conjunction with --report-trx.
From a CLI tool's perspective, this is very reasonable, but because the CLI argument list is the only way to customize those values, it now becomes impossible to just change the format of the trx filenames without having to introduce the extra parameters and without forcing the entire command to always run.
Rather, what we really wanted, was to be able to customize just the format of the filename without touching anything else, and we wanted it to apply only when actually requesting a trx report, which we were trying to setup only in our GitHub workflows, not locally.
Yes, it is possible to add a MSBuild Condition to that property with something like:
<TestingPlatformCommandLineArguments Condition="'$(GITHUB_ACTIONS)' == 'true'">--report-trx --report-trx-filename $(MSBuildProjectName).trx</TestingPlatformCommandLineArguments>
But at the end of the day, we decided against that as it was a bit too obscure. We opted for the less-than-ideal approach of "always generate the trx whenever running dotnet test" even though our need was only to do it on server builds.
The fact that it is all cmdline-based also prevents us from attempting something like this, which would be a much cleaner way of handling this (pseudo-code):
<TestingPlatformCommandLineArguments Condition="'$(cmdline_args).Include('--report-trx')">--report-trx-filename $(MSBuildProjectName).trx</TestingPlatformCommandLineArguments>
Because msbuild does not have access to the passed-in cmdline arguments to dotnet test.
We think this would be significantly simpler if there was a dedicated MSBuild property that only concerned itself with the default format of the trx filenames, so that we could set only that, like:
<TrxFileNameFormat>$(MSBuildProjectName)</TrxFileNameFormat>
And then, whenever --report-trx was used, it would just honor this value automatically, so we could be more explicit on the build server using:
dotnet test -- --report-trx
And avoid the unnecessary process from just happening locally for everyone.
Proposed Feature
I propose that a dedicated MSBuild property be created to customize just the format of the filename for .trx files created by Microsoft.Testing.Extensions.TrxReport.
Suggested property name: TrxFileNameFormat
The commandline value, if specified, would still have precedence over this variable, since it is the more direct/explicit setting.
Alternative Designs
I don't see any other approaches that would be as clean as a dedicated MSBuild property for this. This approach allows using the full breadth of options that MSBuild provides including project-specific properties (such as the one we used, $(MSBuildProjectName)) to customize the format in a generic fashion especially in conjunction with a localized Directory.Build.props file.
Such customization is not possible directly at the commandline when we are running dotnet test because that is usually fired in a much broader context (we are calling it with no solution/project arguments so it grabs the default master solution of the entire repo).
At the same time, the existing TestingPlatformCommandLineArguments also doesn't allow us to cleanly override individual aspects of the generation such as the trx filename without forcing us into a whole other set of decisions.
I think this is preferred to other options such as the one described here:
- https://github.com/microsoft/testfx/issues/5364
Although I could see both working in conjunction as well as they feel like independent enhancements.
I don't think it's a good idea to attempt to let the platform read MSBuild properties at runtime.
The only viable way I can think of to do this is by using RuntimeHostConfigurationOption.
TestingPlatformCommandLineArguments is kinda different. It's not read by MTP at runtime at all. It's simply a way to adjust how the exe is run in the first place, so the exe starts right away with the expected command-line arguments, and then MTP just reads those arguments.
Currently, TestingPlatformCommandLineArguments is the recommended way if you want to adjust TRX name.
@Youssef1313 if msbuild properties is not feasible directly, there must be some other way of handling this. What about propagating environment variables to the MTP executable and customize these values through that?
Currently,
TestingPlatformCommandLineArgumentsis the recommended way if you want to adjust TRX name.
@Youssef1313 what if I need to run my tests using dotnet test --test-modules?
I just realized that using that approach, none of the values declared in TestingPlatformCommandLineArguments are honored at all, and we are back to square one: we need to run multiple test dll's, but we can't customize each output trx file based on the executing project, since the CLI has no context of that.
Now we are kinda stuck with the terrible default names of trx files 😞
My assumption is that something like this:
- https://github.com/microsoft/testfx/issues/5364
Would resolve my problem.