Incorrect warnings logged in diagnostics ".host" file for nunit tests

[bachowny]

I work on some NUnit tests which we run under dotnetcore via vstest.

I've turned on some logging to investigate an issue with the test suite aborting with no message (that's covered elsewhere so I won't go into details here).

So we run the tests like so:

dotnet vstest MontrealTests.dll /Logger:"trx;LogFileName=RESULTS.trx" /Diag:"TestResults/Diagnostics.txt;TraceLevel=Warning"

What I'm noticing though is that we also get a file created called e.g. Diagnostics.host.19-01-18_12-46-24_25406_5.txt. I believe this file is created by NUnit's adapter which is why I'm posting here.

Inside this file, for every single test in my combinatorial tests (which is quite a lot of them), I get a message like this:

TpTrace Warning: 0 : 15036, 30, 2019/01/18, 12:53:45.800, 839984326897, testhost.dll, TestRunCache: No test found corresponding to testResult 'PlaceOrderAndCancel(Bid,XMOD:CGBZ18CGBH19:x-1) Passed' in inProgress list.

TpTrace Warning: 0 : 15036, 30, 2019/01/18, 12:53:46.462, 839986519777, testhost.dll, InProgressTests is null

To be clear, there is nothing wrong with the tests or the results file that is generated. Those are working and the intermittent failure I'm investigating is a separate issue.

This spammy warning though is quite annoying and means that I get several megabytes of diagnostic logs even when nothing has gone wrong.

It would be great if I could suppress this warning or if we just stopped logging it when it's clearly incorrect.

Thanks in advance for any assistance!

[CharliePoole]

Are you using any nunit features to tailor the names of your tests? Are parts of those test names randomly generated?

[bachowny]

Are you using any nunit features to tailor the names of your tests? Are parts of those test names randomly generated?

No the names of the tests are simply the name of the test function and the parameter values for that run - the default NUnit behaviour as far as I know.

[CharliePoole]

Are any of those parameters randomly generated?

[bachowny]

Are any of those parameters randomly generated?

No they're just a list of hardcoded values, sourced from a function using the ValueSource attribute, like this:

public void PlaceOrderAndCancel([ValueSource("StaticBidOrAskValues")] BidOrAsk aBidOrAsk, [ValueSource("StaticInstrumentSeedPairs")] InstrumentSeedPricePair aInstrumentAndPrice)

One thing we are doing that might be a bit strange is we are working around the fact that we can't override static functions with inheritance like this:

    public abstract partial class DepthTestsNew<T> : MarketDataTest where T : DepthTestsNew<T>, new()
    {
      private static T mParameterSource = new T();

      public abstract IEnumerable<InstrumentSeedPricePair> InstrumentSeedPairs();
      public static IEnumerable<InstrumentSeedPricePair> StaticInstrumentSeedPairs()
      {
        return mParameterSource.InstrumentSeedPairs();
      }

      ...
    }

[mikkelbu]

The warnings come from vstest, see https://github.com/Microsoft/vstest/blob/7b6248203164f8ea821f6795632bd22e0b69afb0/src/Microsoft.TestPlatform.CrossPlatEngine/Execution/TestRunCache.cs#L397 and https://github.com/Microsoft/vstest/blob/7b6248203164f8ea821f6795632bd22e0b69afb0/src/Microsoft.TestPlatform.CrossPlatEngine/Execution/TestRunCache.cs#L262, and will - as far as I can tell be reported when you run with /Diag.

Why the warnings are reported I cannot tell without examining it in more detail.

[bachowny]

Should we (can we) move this issue to that project then?

[mikkelbu]

I don't have the right to transfer issues in this repository, and even if I did, we don't have right to move issues to vstest.

So I think that the only option is to create a new issue in vstest, direct link: create issue, and e.g. refer to this issue.

[CharliePoole]

@mikkelbu From this side, should it be transferred to the adapter repo?

BTW I thought all committers could transfer issues.

[mikkelbu]

@CharliePoole I could be transfered there 👍

I think I can transfer using zenhub, but using the beta variant in GitHub I can only do it in nunit.analyzers and not in this project nor in e.g. nunit-console.

[CharliePoole]

I'll do it when I get to my computer if no-one beats me to it.

[OsirisTerje]

@mikkelbu Will you raise the issue in the vstest repo? Once that is done, I'll inform the PG, and refer back to this issue here, and the upcoming vstest issue.

[mikkelbu]

@OsirisTerje Issue raised as microsoft/vstest#1894. I don't know the abbreviation PG in this context?

[OsirisTerje]

Thanks Mikkel! PG = Product Group, which is the Microsoft PG responsible for this. I'm always trying to alert them if there are something they need to look at, also, tagging with the VS Issue is something they now are following up on.