Why TestCase with "\uffff" argument doesn't work?

[Denis535]

[TestCase( "\uffff" )]
public void Test_01(string value) {
    TestContext.WriteLine( value );
}

After running this test stays white. Neither green nor red, just white. Why? It is very strange.

[CharliePoole]

@Denis535 Perhaps you could give some info about what version of the NUnit framework you are using, what runtime you are targeting and what software you use to run the tests?

[mikkelbu]

From the text I guess that you are running the tests from VS, and VS have problems with special names. I think I've seen similar problems in the adapter project before.

[Denis535]

NUnit 3.12.0 NUnit3TestAdapter 3.15.1 Visual Studio 2019 Version 16.7.0 Preview 2.0

It is strange that any other value is displayed as a square. But "\uffff" is displayed as it is.

[mikkelbu]

@Denis535 This is a problem in the adapter, but if you update to 3.16.0, then it should work, if I read nunit/nunit3-vs-adapter#484 correctly. I'll transfer this issue to adapter repo, if it is still a problem in 3.16.0, otherwise I'll close this issue as solved.

[Denis535]

Hmm... My Package source wasn't All. So, I didn't know that there is newer version)) But in 3.16.1 it still doesn't work ((

[mikkelbu]

@Denis535 I've transfered the issue to the adapter repo

[OsirisTerje]

@Denis535 @mikkelbu Yes, this is part of the FQN issues, of which we have multiple. We're currently blocked for most of these issues. We can fix a few, but not all. If you have tried 3.16.0 and it doesn't work, and also 3.16.1 doesn't work, you're in that group. I'll add this one to the growing list of FQN issues.

UPDATE:

I am not so sure this is a FQN Issue. I added the test to a FQN suite, to be found here: https://github.com/nunit/nunit3-vs-adapter.issues/blob/master/Issue640/Issue761.cs

The Results are: As can be seen , the tests with the other Unicode values work, it is only the one with the value \uffff that doesn't.
If I look in the dump file,

I can't see that testcase either, so it is not returned from NUnit either. So this error may be in NUnit itself. @mikkelbu ?

UPDATE 2: Just checked the data coming from the Test Explorer, and also what it sends down to the engine/framework, and all cases are there:

So this seems to be a problem in the engine/framework.
@ChrisMaddock @mikkelbu Can you have a look and move back to either the engine or the framework?

[ChrisMaddock]

I think this is a duplicate of https://github.com/nunit/nunit/issues/3063#issuecomment-432788447 - would you agree?

[OsirisTerje]

Might be yes...

I checked this with the NUnit.Console now, and that one finds ALL test cases here:

So then I am a bit stumped. Not sure why the adapter gets a different result than the console. And, I don't see any exceptions in the output here either

[ChrisMaddock]

Oh, that's odd.

Can you filter with the console? (Using whatever filter the adapter would normally do)

[OsirisTerje]

I just run

nunit3-console fqn.dll

No filter

[ChrisMaddock]

So, if I'm reading the adapter code right, the adapter's dump is built up by combining all the received test events?

That's a different from the console TestResult.xml, which is passed back from the framework in one piece. That suggests there might be an issue with the TestEvent writing functionality in the framework, which isn't present in the xml writing for the final result.

[OsirisTerje]

Not quite. We first do a discovery during the execution, and there we get a complete list of the testcases, just the same as the the first part of the testresult.xml. Then we run the execution, which builds it up using the events The dump you see above is from the first phase, the execution discovery phase

[OsirisTerje]

E_FQN.dll.zip

This is the complete dump file. You see both phases here, there are markers between them

[ChrisMaddock]

Ah ok. So if you run nunit3-console fqn.dll --explore, do you get the test listed?

(Sorry - there's a limited amount I can test myself on this tablet!)

[OsirisTerje]

Yes

It seems to disappear between the engine and the adapter. The dump is coming directly from the

  var discoveryResults = NUnitEngineAdapter.Explore(filter);
                executionDumpXml?.AddString(discoveryResults.AsString());

The Explore method here is just a simple wrapper around:

And then this:

So there should be no magic here... The only possibility is the AsString, but it doesn't balk anywhere else.

[ChrisMaddock]

I'm a little suspicious of AsString - only because it's the only notable difference I can see anywhere! We're also dealing with an invalid XML character being encoded in Xml (I think), so I wouldn't be too surprised if we encountered some strange behaviour somewhere along the way...

Are you able to breakpoint in the debugger, and examine discoveryResults before it goes into AsString()?

[OsirisTerje]

Yep, did that now, and this output is directly from the runner.Explore

<test-suite type="ParameterizedMethod" id="0-1024" name="Test_01" fullname="FQN.Issue761.Test_01" classname="FQN.Issue761" runstate="Runnable" testcasecount="2"><test-case id="0-1010" name="Test_01(&quot;￰&quot;)" fullname="FQN.Issue761.Test_01(&quot;￰&quot;)" methodname="Test_01" classname="FQN.Issue761" runstate="Runnable" seed="1850039790" /><test-case id="0-1012" name="Test_01(&quot;ᄑ&quot;)" fullname="FQN.Issue761.Test_01(&quot;ᄑ&quot;)" methodname="Test_01" classname="FQN.Issue761" runstate="Runnable" seed="340842816" /></test-suite>

Ad there ire only two test cases there......, one is missing

[OsirisTerje]

@ChrisMaddock You did mention some time back something about using the api and not the engine directly, that there are some differences. I don't remember if we did something about that or not. We had issues, which I also don't exactly remember about the ability to use extensions. I think that is still open. Can there be something there about the results we get abck, that we're using the wrong interface somehow ?

[ChrisMaddock]

That was my go-to thought to be honest, but from looking at how the adapter does use the engine, I can't think of anything that would be a problem around this...

I'll see if I can get a chance to have a look at this over the weekend.

[ChrisMaddock]

I've ran this in the debugger, I think this is a duplicate of nunit/nunit#3063.

I'm not too sure what you were seeing last night, Terje - when I run the tests, the \ufff test is discovered fine. It's only on execution that we don't get a result, which is consistent with the above issue and the fact that \ufff can't be constructed into a valid filter for a test name at the moment. @jnm2 explains the problems nicely on that issue.

Here's my discovery dump file: D_FQN.dll.txt

[ChrisMaddock]

FWIW, you get the equivalent effect if you try and filter on the test in the Console.

"C:\Users\Chris\Documents\git\nunit3-vs-adapter.issues\Issue640\bin\Debug\net472\FQN.dll" --test FQN.Issue761.Test_01("\uffff")

[OsirisTerje]

Thanks @ChrisMaddock . @jnm2 suggest filtering these in the adapter. Can do that, but as long as the nunit issue remains, it will still be broken. Changing the name in the adapter never works well.
.---- and, The test is skipped during execution.

[dannypike]

I'm running NUnit Adapter 3.17.0.0, on "Visual Studio 16.7.0 Preview 3.1" and I get the same problem with "\u0001", "\u0010" and "\u0083" but it works OK with "\u0007".

Wondering out loud: might it it have anything specific to do with Unicode-only control characters - i.e. char.IsControl() returns true - but where they do not have equivalent ASCII escape code sequences?

For example, the value "\u0007" above can also be written as "\a", and codes like "\u000d" and "\u000a" also map directly to ASCII codes "\r" and "\n". All of those work, but the ones that have no dedicated ASCII escape codes seem to fail.

[OsirisTerje]

Thanks @dannypike , interesting observation.

[felipepessoto]

I have the same issue. Unicode chars are pretty common test case argument in unit tests, I think this is high pri since tests are just ignored and you can easily miss it from CI pipeline.

    [TestCase("\u0006")]
    [TestCase("\u009A")]
    [TestCase("\u009B")]
    [TestCase("\u009C")]
    [TestCase("\u009D")]
    [TestCase("\u009E")]
    [TestCase("\u009F")]
    [TestCase("\u009C")]
    [TestCase("OK")]
    public void TestUnicode(string value)
    {
        if(value == "\u009B")
        {
            throw new System.Exception("ABC");
        }
        Assert.Pass();
    }

Probably the problem is in nunit and not in the adapter, because running

nunit3-console.exe "TestProject1.dll" --test TestProject1.Tests.TestUnicode(\"\u009b\")

Show it correctly receive the test name, but find 0 tests:

If I run it using --test TestProject1.Tests.TestUnicode("OK"), it works:

[felipepessoto]

Some notes about the bug. It basically happens because XML filter contains special chars in the \u format while the filter compares it to the actual special char

If I debug and remove one of slashes, to un-escape it (NUnit3FrameworkDriver.CreateObject), it works:

In the ValueMatchFilter class, the expected value contains the parameter in the \u format (which came from TestFilter.FromXml() which gets it from xml string which is in the escaped format), while the input (created at NUnitTestCaseBuilder.BuildTestMethod) is not escaped and contains the actual char: