Profile Test gives error "Illegal characters in path."

[xavierjohn]

Right-clicking any xunit unit test and clicking "Profile Test" gives error "Illegal characters in path." Visual Studio 2017 (15.3.5)

You can try any xunit tests even the ones on https://github.com/xavierjohn/super-funicular

[bradwilson]

Profiling does not support .NET Core at this time (regardless of test framework).

Changing the target framework of your project to net452 allows profiling to work as expected.

[xavierjohn]

The problem persists after switching to net452.

I switched C:\Git\github\xavier\super-funicular\SuperFuncular\SuperFuncular\Matrix\SparseColumnMatrix.cs to use mstest and profiling works.

[bradwilson]

It's possible that MSTest is probably profiling the code using the desktop CLR rather than .NET Core; we will not do that, because load issues would prevent it.

I am not able to reproduce your problem when switching to net452.

[xavierjohn]

I pushed out my changes to switch to net452 and “Profile Test” on SubStringPerformanceTest fails with the same error “[10/4/2017 6:57:57 PM Error] Illegal characters in path.”

I can’t see any difference between our setup. I am able to reproduce on two independent machines.

Sent from Mailhttps://go.microsoft.com/fwlink/?LinkId=550986 for Windows 10

From: Brad Wilsonmailto:[email protected] Sent: Wednesday, October 4, 2017 12:32 PM To: xunit/xunitmailto:[email protected] Cc: Xaviermailto:[email protected]; Authormailto:[email protected] Subject: Re: [xunit/xunit] Profile Test gives error "Illegal characters in path." (#1483)

It's possible that MSTest is probably profiling the code using the desktop CLR rather than .NET Core; we will not do that, because load issues would prevent it.

I am not able to reproduce your problem when switching to net452.

[image]https://user-images.githubusercontent.com/16944/31195677-ea1842c4-a8ff-11e7-98f6-32ae2ac8f09e.png

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHubhttps://github.com/xunit/xunit/issues/1483#issuecomment-334264971, or mute the threadhttps://github.com/notifications/unsubscribe-auth/ABxgfhXhKherJ27K88R7r1qhXJxZLpQaks5so909gaJpZM4Ps5oQ.

[bradwilson]

I get a different error when trying to profile it with .NET Core:

Maybe you actually have two separate problems: the fact that Visual Studio can't profile .NET Core AND whatever this "illegal characters in path" problem is.

[commonsensesoftware]

I was able to reproduce the behavior using only .NET 4.5.2. No .NET Core and no multi-targeting. I'm using Visual Studio 2017 (15.3.5).

Additional Discoveries

• The issue only seems occur for theories; facts were profiled just fine for me
• If I disable the pre-enumerate theories feature (which is true by default), the issue appears to go away

Workaround

Not pre-enumerating theories seems to do the trick, but may not be how a dev wants to use xUnit. I don't fully understand how theory test cases match to VSTest. I would presume it would be using the type and method info from the enumerated test case. For some reason, VSTest doesn't seem to like the theory test cases generated by xUnit for profiling. I suspect the culprit is more on the VSTest side than the xUnit side, but I'm not sure how you can prove that.

Next Steps

@bradwilson are there logs or something else that can use to prove the issue is on the VSTest side? Do you happen to have a contact or issue log were this can be posted to the VS team (offline if necessary)? Without a clear hand-off to VS, I'm afraid there will be just a bunch of back and forth finger pointing. I doubt this is an xUnit issue.

[bradwilson]

Can you provide a repro project? I've not been able to repro this myself.

[commonsensesoftware]

With the latest updates to Visual Studio and XUnit, I noticed there are a few twists to the conditions. The following outlines as much detail as I can with a repro.

Setup

• Visual Studio 2017 (15.4.3)
• XUnit (2.3.1)

NOTE: I recommend deleting all folders in %TEMP%\VisualStudioTestExplorerExtensions and restart Visual Studio before starting

Example test cases:

public class TestCases
{
    [Fact]
    public void any_fact() => Assert.True(true);

    [Theory]
    [InlineData("1")]
    [InlineData("2")]
    [InlineData("3")]
    public void any_theory(string input) => Assert.NotNull(input);
}

Repro Steps

1. Build the solution
2. Run the tests
3. Right-click any enumerated test case for any_theory in the Test Explorer and select Profile Test

Expected Result Instrumentation profile report for any_theory is created and opened.

Actual Result If any_theory is the first test case profiled in a new instance of Visual Studio, you get "Illegal characters in path". If any_theory is the Nth test case profiled after any_fact has successfully been generated, the instrumentation profile report for any_fact is created and opened instead of any_theory.

Workaround Repro Steps

1. Change preEnumerateTheories to false in xunit.runner.json
2. Rebuild the solution
3. Run the tests
4. Right-click any_theory in the Test Explorer and select Profile Test

Actual Result Instrumentation profile report for any_theory is created and opened as expected.

Repro Artifacts

I've attached a repro project and a few screenshots that will hopefully provide additional information useful to you. xunit-issue-1483-repro.zip

Figure 1: Incorrect instrumentation profile report is generated

Figure 2: Correct instrumentation profile report is generated if theories are not pre-enumerated

Figure 3: Illegal characters in path if pre-enumerated theory is the first test profiled

[bradwilson]

I feel like, based on this description, that it's going to end up being a case of VSTest not correctly support pre-enumerated theories in this manner, probably because it cannot pass enough information to the profiler to ensure that it runs just the one test in question (and then gets confused). Have you opened an issue with the VSTest team yet?

[commonsensesoftware]

I agree.

No. I'm not aware of an issue being opened up against the VSTest team. Do you have a link where they track their issues? I think it's worth them saying it's not their issue before going down what will almost certainly be a deep rabbit hole.

[bradwilson]

https://github.com/microsoft/vstest/issues