vstest The active test run was aborted. Reason: Unexpected character encountered

Description

If I pass strings that are not correct Unicode as parameters to xunit theories, the test run fails:

The active test run was aborted. Reason: Unexpected character encountered while parsing value

If I use dotnet test --diag log.log, I see that the values cannot be deserialized by Newtonsoft.Json. I'm not certain whether this is a bug of xunit runner (I've created the issue just in case), or of the runner infrastructure of vstest.

TpTrace Verbose: 0 : 18612, 10, 2020/10/22, 16:31:21.906, 155947063999, vstest.console.dll, TestRequestSender: GetAbortErrorMessage: Exception: Newtonsoft.Json.JsonReaderException: Bad JSON escape sequence: \�. Path 'Payload.NewTestResults[1].DisplayName', line 1, position 1896. at Newtonsoft.Json.JsonTextReader.ReadStringIntoBuffer(Char quote) at Newtonsoft.Json.JsonTextReader.ParseValue() at Newtonsoft.Json.JsonWriter.WriteToken(JsonReader reader, Boolean writeChildren, Boolean writeDateConstructorAsDate, Boolean writeComments) at Newtonsoft.Json.Linq.JTokenWriter.WriteToken(JsonReader reader, Boolean writeChildren, Boolean writeDateConstructorAsDate, Boolean writeComments) at Newtonsoft.Json.JsonWriter.WriteToken(JsonReader reader, Boolean writeChildren) at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateJToken(JsonReader reader, JsonContract contract) at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.SetPropertyValue(JsonProperty property, JsonConverter propertyConverter, JsonContainerContract containerContract, JsonProperty containerProperty, JsonReader reader, Object target) at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.PopulateObject(Object newObject, JsonReader reader, JsonObjectContract contract, JsonProperty member, String id) at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateObject(JsonReader reader, Type objectType, JsonContract contract, JsonProperty member, JsonContainerContract containerContract, JsonProperty containerMember, Object existingValue) at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.Deserialize(JsonReader reader, Type objectType, Boolean checkAdditionalContent) at Newtonsoft.Json.JsonSerializer.DeserializeInternal(JsonReader reader, Type objectType) at Newtonsoft.Json.JsonSerializer.Deserialize[T](JsonReader reader) at Microsoft.VisualStudio.TestPlatform.CommunicationUtilities.JsonDataSerializer.Deserialize[T](JsonSerializer serializer, String data) at Microsoft.VisualStudio.TestPlatform.CommunicationUtilities.JsonDataSerializer.DeserializeMessage(String rawMessage) at Microsoft.VisualStudio.TestPlatform.CrossPlatEngine.Client.ProxyExecutionManager.HandleRawMessage(String rawMessage) at Microsoft.VisualStudio.TestPlatform.CommunicationUtilities.TestRequestSender.OnExecutionMessageReceived(Object sender, MessageReceivedEventArgs messageReceived, ITestRunEventsHandler testRunEventsHandler)

Steps to reproduce

Use dotnet test to test the following xunit code:

using System;
using System.Linq;
using System.Text;
using Xunit;

namespace XUnitRunner
{
    public class UnitTest1
    {
        [Theory]
        [MemberData(nameof(GetRandomStrings))]
        public void Test1(string @string)
        {
            Assert.Equal(@string, "a");
        }

        public static TheoryData<string> GetRandomStrings()
        {
            var data = new TheoryData<string>();

            var generator = new Random();

            char PickRandomCharacter()
                => randomCharacters[generator.Next(0, randomCharacters.Length)];

            foreach(var i in Enumerable.Range(0, 100))
            {
                var randomString = Enumerable.Range(0, 100)
                    .Aggregate(
                        new StringBuilder(),
                        (builder, character) => builder.Append(
                            PickRandomCharacter()))
                    .ToString();

                data.Add(randomString);
            }

            return data;
        }

        private static readonly char[] randomCharacters
            = Enumerable.Range(char.MinValue, char.MaxValue + 1)
                .Select(characterCode => Convert.ToChar(characterCode))
                .Where(c => !char.IsLetterOrDigit(c))
                .ToArray();
    }
}

Expected behavior

All of the tests are being run.

Actual behavior

The test run crashes.

Diagnostic logs

I'm using dotnet test.

Environment

I'm using dotnet test with .Net Core 3.1.402 on Windows 10 (10.0.19041.572).

Oct 22 '20 13:10 quixoticaxis

Based on the callstack it looks like our issue:

at Microsoft.VisualStudio.TestPlatform.CommunicationUtilities.JsonDataSerializer.Deserialize[T](JsonSerializer serializer, String data)

Do you know which character causes that issue specifically?

Nov 02 '20 14:11 nohwnd

Unfortunately, I don't know.

I empirically found out that it definitely fails on a string of the first seven high surrogates (integer codes: 55296, 55297, 55298, 55299, 55300, 55301, 55302):

Enumerable.Range(char.MinValue, char.MaxValue + 1)
    .Select(characterCode => Convert.ToChar(characterCode))
    .Where(c => char.IsHighSurrogate(c))
    .Take(7)

and it fails with the error:

The active test run was aborted. Reason: Unexpected character encountered while parsing value: ?.

I can't find a particular sequence that leads to the error in my original post. That's why I pick strings randomly.

Nov 02 '20 18:11 quixoticaxis

@quixoticaxis thanks for trying, too bad you are not using seed for the random, that way this would have been reproducible. :(

Nov 30 '20 15:11 nohwnd

@nohwnd As I've mentioned above, it can be reproduced when using first seven high surrogates (no randomness at all). It can be also reproduced when using the seed of 1 (new Random(1) at line 21). The interesting thing is that the runner no longer fails if the project is switched from netcoreapp3.1 to net5.0.

Nov 30 '20 16:11 quixoticaxis

I confirm that the error is still present:

Microsoft (R) Test Execution Command Line Tool Version 17.3.0-preview-20220414-05 (x64)
Copyright (c) Microsoft Corporation.  All rights reserved.

Starting test execution, please wait...
A total of 1 test files matched the specified pattern.
[xUnit.net 00:00:00.49]     XUnitRunner.UnitTest1.Test1(string: "\xd8e2\xd80fྀ໶㊙⣻㋯꩞⍄᫽⯹᫯╙⛡ഁ⑊㋑꓉⵼∊\xdcbd\xdc0e੟"...) [FAIL]
[xUnit.net 00:00:00.50]     XUnitRunner.UnitTest1.Test1(string: "⼶⥴\xded3\xd9f1᭏﫠⟭⫍፵ۜ♓໢⾽⍜੸⒧“ꪰ\xdb18໌ᷬ㄄\xde29"...) [FAIL]
[xUnit.net 00:00:00.50]     XUnitRunner.UnitTest1.Test1(string: "\xda34⠍ᳳ㏼⣩Ⓣ؈⩺⣊\xdfc4➑：⊈౱⭡⡼∰\xde7d༗㈛㌰⵫➷꣊︁"...) [FAIL]
[xUnit.net 00:00:00.50]     XUnitRunner.UnitTest1.Test1(string: "ⅷ∨╘᥺೰〔\xdac3\xdd79\xddb9〉\xda18☏ᩝ\xd"...) [FAIL]
[xUnit.net 00:00:00.50]     XUnitRunner.UnitTest1.Test1(string: "=\xd9f1᧳ᷬ\xdf3a⹆૴̊⋿◈꬛⏝㎘ⅹ⊼㈷\xda1b"...) [FAIL]
[xUnit.net 00:00:00.50]     XUnitRunner.UnitTest1.Test1(string: "\xdf07↎৉\xdae7¶˙▪੎ꫭᜲ㉃ૈ\xdca3⅛ⅲ⧲\xdead⍅\xd"...) [FAIL]
[xUnit.net 00:00:00.50]     XUnitRunner.UnitTest1.Test1(string: "┰᜴੿✾\x13͆﷥⨂⬽☑\xd8ccܿ\xdd00\xd8e7䶿\xda3"...) [FAIL]
[xUnit.net 00:00:00.50]     XUnitRunner.UnitTest1.Test1(string: "⃲൹࿘☪⹘᳨\xd894\xdafa⫛„೥⻭౻⛖⻸ᤩ́⁔ⷢ񼽞\xdc3f⬙"...) [FAIL]
[xUnit.net 00:00:00.51]     XUnitRunner.UnitTest1.Test1(string: "⟪⿃㉃⎹\xdcf5\xd8db﯄﹍኿䷽\xde76䶷⪘㈉㌁\xdd4c᠟ᩩ"...) [FAIL]
The active test run was aborted. Reason: After parsing a value an unexpected character was encountered: �. Path 'Payload.NewTestResults[8]', line 1, position 9755.

Test Run Aborted with error System.Exception: One or more errors occurred.
 ---> System.Exception: After parsing a value an unexpected character was encountered: �. Path 'Payload.NewTestResults[8]', line 1, position 9755.
   at Newtonsoft.Json.JsonTextReader.ParsePostValue(Boolean ignoreComments)
   at Newtonsoft.Json.JsonTextReader.Read()
   at Newtonsoft.Json.JsonWriter.WriteToken(JsonReader reader, Boolean writeChildren, Boolean writeDateConstructorAsDate, Boolean writeComments)
   at Newtonsoft.Json.JsonWriter.WriteToken(JsonReader reader, Boolean writeChildren)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateJToken(JsonReader reader, JsonContract contract)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateValueInternal(JsonReader reader, Type objectType, JsonContract contract, JsonProperty member, JsonContainerContract containerContract, JsonProperty containerMember, Object existingValue)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.SetPropertyValue(JsonProperty property, JsonConverter propertyConverter, JsonContainerContract containerContract, JsonProperty containerProperty, JsonReader reader, Object target)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.PopulateObject(Object newObject, JsonReader reader, JsonObjectContract contract, JsonProperty member, String id)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateObject(JsonReader reader, Type objectType, JsonContract contract, JsonProperty member, JsonContainerContract containerContract, JsonProperty containerMember, Object existingValue)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateValueInternal(JsonReader reader, Type objectType, JsonContract contract, JsonProperty member, JsonContainerContract containerContract, JsonProperty containerMember, Object existingValue)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.Deserialize(JsonReader reader, Type objectType, Boolean checkAdditionalContent)
   at Newtonsoft.Json.JsonSerializer.DeserializeInternal(JsonReader reader, Type objectType)
   at Newtonsoft.Json.JsonSerializer.Deserialize[T](JsonReader reader)
   at Microsoft.VisualStudio.TestPlatform.CommunicationUtilities.JsonDataSerializer.Deserialize[T](JsonSerializer serializer, String data)
   at Microsoft.VisualStudio.TestPlatform.CrossPlatEngine.Client.ProxyExecutionManager.HandleRawMessage(String rawMessage)
   at Microsoft.VisualStudio.TestPlatform.CommunicationUtilities.TestRequestSender.OnExecutionMessageReceived(MessageReceivedEventArgs messageReceived, ITestRunEventsHandler testRunEventsHandler)
   --- End of inner exception stack trace ---.
▓   amauryleveLAPTOP-UC6GDM3J   C:\src\vstest   netcoreapp3.1 ≡  11

Jul 25 '22 14:07 Evangelink

I'm getting slightly different error

Test Run Aborted with error System.Exception: One or more errors occurred.
 ---> System.Exception: Unexpected character encountered while parsing value: �. Path '', line 0, position 0.
   at Newtonsoft.Json.JsonTextReader.ParseValue()
   at Newtonsoft.Json.JsonReader.ReadForType(JsonContract contract, Boolean hasConverter)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.Deserialize(JsonReader reader, Type objectType, Boolean checkAdditionalContent)
   at Newtonsoft.Json.JsonSerializer.DeserializeInternal(JsonReader reader, Type objectType)
   at Microsoft.VisualStudio.TestPlatform.CommunicationUtilities.JsonDataSerializer.DeserializeObjectFast[T](String value) in /_/src/Microsoft.TestPlatform.CommunicationUtilities/JsonDataSerializer.cs:line 230
   at Microsoft.VisualStudio.TestPlatform.CommunicationUtilities.JsonDataSerializer.DeserializeMessage(String rawMessage) in /_/src/Microsoft.TestPlatform.CommunicationUtilities/JsonDataSerializer.cs:line 145
   at Microsoft.VisualStudio.TestPlatform.CrossPlatEngine.Client.ProxyExecutionManager.HandleRawMessage(String rawMessage) in /_/src/Microsoft.TestPlatform.CrossPlatEngine/Client/ProxyExecutionManager.cs:line 437
   at Microsoft.VisualStudio.TestPlatform.CommunicationUtilities.TestRequestSender.OnExecutionMessageReceived(MessageReceivedEventArgs messageReceived, IInternalTestRunEventsHandler testRunEventsHandler) in /_/src/Microsoft.TestPlatform.CommunicationUtilities/TestRequestSender.cs:line 538
   --- End of inner exception stack trace ---.

Apr 19 '24 21:04 olstakh

Tried using the latest dotnet test in net9, and it works as expected, probably because of the escaping fixes we did for F# and single quote. 🤔

Jul 10 '24 11:07 nohwnd

I tried randomly shuffling char sequnces and, yeap, now the tests fail like they should (tested on millions of inputs) without crashing.

Jul 11 '24 00:07 quixoticaxis

I'm getting slightly different error

Test Run Aborted with error System.Exception: One or more errors occurred.
 ---> System.Exception: Unexpected character encountered while parsing value: �. Path '', line 0, position 0.
   at Newtonsoft.Json.JsonTextReader.ParseValue()
   at Newtonsoft.Json.JsonReader.ReadForType(JsonContract contract, Boolean hasConverter)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.Deserialize(JsonReader reader, Type objectType, Boolean checkAdditionalContent)
   at Newtonsoft.Json.JsonSerializer.DeserializeInternal(JsonReader reader, Type objectType)
   at Microsoft.VisualStudio.TestPlatform.CommunicationUtilities.JsonDataSerializer.DeserializeObjectFast[T](String value) in /_/src/Microsoft.TestPlatform.CommunicationUtilities/JsonDataSerializer.cs:line 230
   at Microsoft.VisualStudio.TestPlatform.CommunicationUtilities.JsonDataSerializer.DeserializeMessage(String rawMessage) in /_/src/Microsoft.TestPlatform.CommunicationUtilities/JsonDataSerializer.cs:line 145
   at Microsoft.VisualStudio.TestPlatform.CrossPlatEngine.Client.ProxyExecutionManager.HandleRawMessage(String rawMessage) in /_/src/Microsoft.TestPlatform.CrossPlatEngine/Client/ProxyExecutionManager.cs:line 437
   at Microsoft.VisualStudio.TestPlatform.CommunicationUtilities.TestRequestSender.OnExecutionMessageReceived(MessageReceivedEventArgs messageReceived, IInternalTestRunEventsHandler testRunEventsHandler) in /_/src/Microsoft.TestPlatform.CommunicationUtilities/TestRequestSender.cs:line 538
   --- End of inner exception stack trace ---.

I'm still getting this, although the OP scenario does not reproduce crash. I'll try upgrading xunit to the latest and see if it still hits.

Meanwhile, @nohwnd , is there a reason why --blame-crash switch doesn't work for this type of crash? I see dumps generated from other crashes, but not from this Unexpected character one. Basically if i downgrade xunit to the version where i can repro OP's scenario - i won't get memory dumps with --blame-crash, is this an issue i should open separately, or there's an explanation as to why its by design?

Jul 12 '24 19:07 olstakh

I'm getting slightly different error

Can you share the test which leads to the error?

Jul 13 '24 14:07 quixoticaxis

Also, IMHO, escaping is not an ideal way to go. Json defines strings as sequences of code points, while .Net code treats strings as sequences of code units. IMHO, .Net strings should be treated as binary throughout test infrastructure.

Jul 13 '24 14:07 quixoticaxis