diagnostics icon indicating copy to clipboard operation
diagnostics copied to clipboard
verified publisher icon dotnet

BasicProcessInfoNoSuspendTest and BasicProcessInfoNoSuspendTestAsync are flaky

Open hoyosjs opened this issue 4 years ago • 7 comments

hoyosjs avatar Nov 23 '21 00:11 hoyosjs

Adding BasicProcessInfoSuspendTest to the list of flaky tests. Here is the stack trace:

at System.IO.BinaryReader.InternalRead(Int32 numBytes) at System.IO.BinaryReader.ReadUInt16() at Microsoft.Diagnostics.NETCore.Client.IpcHeader.Parse(BinaryReader reader) in //src/Microsoft.Diagnostics.NETCore.Client/DiagnosticsIpc/IpcHeader.cs:line 56 at Microsoft.Diagnostics.NETCore.Client.IpcMessage.Parse(Stream stream) in //src/Microsoft.Diagnostics.NETCore.Client/DiagnosticsIpc/IpcMessage.cs:line 116 at Microsoft.Diagnostics.NETCore.Client.IpcClient.Read(Stream stream) in //src/Microsoft.Diagnostics.NETCore.Client/DiagnosticsIpc/IpcClient.cs:line 107 at Microsoft.Diagnostics.NETCore.Client.IpcClient.SendMessageGetContinuation(IpcEndpoint endpoint, IpcMessage message) in //src/Microsoft.Diagnostics.NETCore.Client/DiagnosticsIpc/IpcClient.cs:line 44 at Microsoft.Diagnostics.NETCore.Client.DiagnosticsClient.TryGetProcessInfo2() in //src/Microsoft.Diagnostics.NETCore.Client/DiagnosticsClient/DiagnosticsClient.cs:line 346 at Microsoft.Diagnostics.NETCore.Client.DiagnosticsClient.GetProcessInfo() in //src/Microsoft.Diagnostics.NETCore.Client/DiagnosticsClient/DiagnosticsClient.cs:line 318 at Microsoft.Diagnostics.NETCore.Client.DiagnosticsClientApiShim.GetProcessInfo(TimeSpan timeout) in //src/tests/Microsoft.Diagnostics.NETCore.Client/DiagnosticsClientApiShim.cs:line 50 at Microsoft.Diagnostics.NETCore.Client.GetProcessInfoTests.BasicProcessInfoTestCore(Boolean useAsync, Boolean suspend) in //src/tests/Microsoft.Diagnostics.NETCore.Client/GetProcessInfoTests.cs:line 62 --- End of stack trace from previous location ---

mikelle-rogers avatar Apr 28 '22 22:04 mikelle-rogers

BasicProcessInfoSuspendTestAsync

System.IO.EndOfStreamException : Attempted to read past the end of the stream.

at Microsoft.Diagnostics.NETCore.Client.StreamExtensions.ReadBytesAsync(Stream stream, Int32 length, CancellationToken cancellationToken) in //src/Microsoft.Diagnostics.NETCore.Client/StreamExtensions.cs:line 26 at Microsoft.Diagnostics.NETCore.Client.IpcHeader.ParseAsync(Stream stream, CancellationToken cancellationToken) in //src/Microsoft.Diagnostics.NETCore.Client/DiagnosticsIpc/IpcHeader.cs:line 70 at Microsoft.Diagnostics.NETCore.Client.IpcMessage.ParseAsync(Stream stream, CancellationToken cancellationToken) in //src/Microsoft.Diagnostics.NETCore.Client/DiagnosticsIpc/IpcMessage.cs:line 125 at Microsoft.Diagnostics.NETCore.Client.IpcClient.SendMessageGetContinuationAsync(IpcEndpoint endpoint, IpcMessage message, CancellationToken cancellationToken) in //src/Microsoft.Diagnostics.NETCore.Client/DiagnosticsIpc/IpcClient.cs:line 83 at Microsoft.Diagnostics.NETCore.Client.DiagnosticsClient.TryGetProcessInfo2Async(CancellationToken token) in //src/Microsoft.Diagnostics.NETCore.Client/DiagnosticsClient/DiagnosticsClient.cs:line 353 at Microsoft.Diagnostics.NETCore.Client.DiagnosticsClient.GetProcessInfoAsync(CancellationToken token) in //src/Microsoft.Diagnostics.NETCore.Client/DiagnosticsClient/DiagnosticsClient.cs:line 332 at Microsoft.Diagnostics.NETCore.Client.DiagnosticsClientApiShim.GetProcessInfo(TimeSpan timeout) in //src/tests/Microsoft.Diagnostics.NETCore.Client/DiagnosticsClientApiShim.cs:line 46 at Microsoft.Diagnostics.NETCore.Client.GetProcessInfoTests.BasicProcessInfoTestCore(Boolean useAsync, Boolean suspend) in //src/tests/Microsoft.Diagnostics.NETCore.Client/GetProcessInfoTests.cs:line 62 --- End of stack trace from previous location where exception was thrown ---

mikelle-rogers avatar Apr 28 '22 22:04 mikelle-rogers

BasicProcessInfoNoSuspendTestAsync

Error message System.IO.EndOfStreamException : Attempted to read past the end of the stream.

Stack trace at Microsoft.Diagnostics.NETCore.Client.StreamExtensions.ReadBytesAsync(Stream stream, Int32 length, CancellationToken cancellationToken) in //src/Microsoft.Diagnostics.NETCore.Client/StreamExtensions.cs:line 23 at Microsoft.Diagnostics.NETCore.Client.IpcHeader.ParseAsync(Stream stream, CancellationToken cancellationToken) in //src/Microsoft.Diagnostics.NETCore.Client/DiagnosticsIpc/IpcHeader.cs:line 70 at Microsoft.Diagnostics.NETCore.Client.IpcMessage.ParseAsync(Stream stream, CancellationToken cancellationToken) in //src/Microsoft.Diagnostics.NETCore.Client/DiagnosticsIpc/IpcMessage.cs:line 125 at Microsoft.Diagnostics.NETCore.Client.IpcClient.SendMessageGetContinuationAsync(IpcEndpoint endpoint, IpcMessage message, CancellationToken cancellationToken) in //src/Microsoft.Diagnostics.NETCore.Client/DiagnosticsIpc/IpcClient.cs:line 83 at Microsoft.Diagnostics.NETCore.Client.DiagnosticsClient.TryGetProcessInfo2Async(CancellationToken token) in //src/Microsoft.Diagnostics.NETCore.Client/DiagnosticsClient/DiagnosticsClient.cs:line 353 at Microsoft.Diagnostics.NETCore.Client.DiagnosticsClient.GetProcessInfoAsync(CancellationToken token) in //src/Microsoft.Diagnostics.NETCore.Client/DiagnosticsClient/DiagnosticsClient.cs:line 332 at Microsoft.Diagnostics.NETCore.Client.DiagnosticsClientApiShim.GetProcessInfo(TimeSpan timeout) in //src/tests/Microsoft.Diagnostics.NETCore.Client/DiagnosticsClientApiShim.cs:line 46 at Microsoft.Diagnostics.NETCore.Client.GetProcessInfoTests.GetProcessInfoWithEntrypointAsync(DiagnosticsClientApiShim shim) in //src/tests/Microsoft.Diagnostics.NETCore.Client/GetProcessInfoTests.cs:line 106 at Microsoft.Diagnostics.NETCore.Client.GetProcessInfoTests.BasicProcessInfoTestCore(Boolean useAsync, Boolean suspend) in /_/src/tests/Microsoft.Diagnostics.NETCore.Client/GetProcessInfoTests.cs:line 71 --- End of stack trace from previous location ---

mikelle-rogers avatar Apr 28 '22 22:04 mikelle-rogers

BasicProcessInfoSuspendTestAsync

Error message System.Threading.Tasks.TaskCanceledException : A task was canceled.

Stack trace at Microsoft.Diagnostics.NETCore.Client.StreamExtensions.ReadBytesAsync(Stream stream, Int32 length, CancellationToken cancellationToken) in //src/Microsoft.Diagnostics.NETCore.Client/StreamExtensions.cs:line 20 at Microsoft.Diagnostics.NETCore.Client.IpcHeader.ParseAsync(Stream stream, CancellationToken cancellationToken) in //src/Microsoft.Diagnostics.NETCore.Client/DiagnosticsIpc/IpcHeader.cs:line 70 at Microsoft.Diagnostics.NETCore.Client.IpcMessage.ParseAsync(Stream stream, CancellationToken cancellationToken) in //src/Microsoft.Diagnostics.NETCore.Client/DiagnosticsIpc/IpcMessage.cs:line 125 at Microsoft.Diagnostics.NETCore.Client.IpcClient.SendMessageGetContinuationAsync(IpcEndpoint endpoint, IpcMessage message, CancellationToken cancellationToken) in //src/Microsoft.Diagnostics.NETCore.Client/DiagnosticsIpc/IpcClient.cs:line 83 at Microsoft.Diagnostics.NETCore.Client.DiagnosticsClient.TryGetProcessInfo2Async(CancellationToken token) in //src/Microsoft.Diagnostics.NETCore.Client/DiagnosticsClient/DiagnosticsClient.cs:line 353 at Microsoft.Diagnostics.NETCore.Client.DiagnosticsClient.GetProcessInfoAsync(CancellationToken token) in //src/Microsoft.Diagnostics.NETCore.Client/DiagnosticsClient/DiagnosticsClient.cs:line 332 at Microsoft.Diagnostics.NETCore.Client.DiagnosticsClientApiShim.GetProcessInfo(TimeSpan timeout) in //src/tests/Microsoft.Diagnostics.NETCore.Client/DiagnosticsClientApiShim.cs:line 46 at Microsoft.Diagnostics.NETCore.Client.GetProcessInfoTests.BasicProcessInfoTestCore(Boolean useAsync, Boolean suspend) in //src/tests/Microsoft.Diagnostics.NETCore.Client/GetProcessInfoTests.cs:line 62 --- End of stack trace from previous location ---

mikelle-rogers avatar Apr 28 '22 22:04 mikelle-rogers

@josalem Do you have a hunch as to what is going on?

mikelle-rogers avatar Apr 28 '22 22:04 mikelle-rogers

These look like two different types of errors:

  1. "Read Past End of Stream"
  2. TaskCanceledException

Both indicate an issue with the target process. In case 1, I'm betting the target process exited unceremoniously either by force or a crash. The latter implies a timing bug in the test harness. If you are trying to triage them further, you'll want to try and repro them locally. If they don't easily repro locally, then that lends credence to this being a timing issue. You could try limiting the test to 2 cores and then straining the cores while the test is running, e.g., run multiple instances of the test at once in a while true loop. If all else fails, open a PR with some logging in smart locations and hope we can catch it in the act in CI with more information.

josalem avatar May 02 '22 16:05 josalem

PR #5354 addresses a timing issue that caused the test to fail with a SocketException saying the connection was refused. This occured because the test was racing calls to GetProcessInfo with starting up the target process. The retries added will hopefully resolve that part of the problem but it doesn't address some of these other reported symptoms which likely have different root causes.

noahfalk avatar Mar 19 '25 06:03 noahfalk