Soulseek.NET icon indicating copy to clipboard operation
Soulseek.NET copied to clipboard

Fix (or remove) flaky tests

Open jpdillingham opened this issue 3 years ago • 1 comments

Test runs in Circle CI fail about half the time on transient errors that don't happen locally. A number of tests rely on timing tricks and waits because some behavior is timer-based, and these are the ones causing the problems.

Review past couple dozen failed CI runs and figure out which tests are causing the most issues. Either fix them or remove them.

jpdillingham avatar Jan 17 '22 15:01 jpdillingham

Starting test execution, please wait...
A total of 1 test files matched the specified pattern.
[xUnit.net 00:00:04.98]     DownloadToFileAsync uses given size when skipping queue(username: "usernameb9e1a1ca-d0ef-4c3d-bad2-e2142b38e06f", endpoint: 127.0.0.1:73, filename: "filenameeb0feca4-2bc4-4eb1-a842-cf0e17c6b045", token: 193, size: 133, givenSize: 167) [FAIL]
  Failed DownloadToFileAsync uses given size when skipping queue(username: "usernameb9e1a1ca-d0ef-4c3d-bad2-e2142b38e06f", endpoint: 127.0.0.1:73, filename: "filenameeb0feca4-2bc4-4eb1-a842-cf0e17c6b045", token: 193, size: 133, givenSize: 167) [9 ms]
  Error Message:
   Soulseek.SoulseekClientException : Failed to download file filenameeb0feca4-2bc4-4eb1-a842-cf0e17c6b045 from user usernameb9e1a1ca-d0ef-4c3d-bad2-e2142b38e06f: Object reference not set to an instance of an object.
---- System.NullReferenceException : Object reference not set to an instance of an object.
  Stack Trace:
     at Soulseek.SoulseekClient.DownloadToStreamAsync(String username, String remoteFilename, Func`1 outputStreamFactory, Nullable`1 size, Int64 startOffset, Int32 token, TransferOptions options, CancellationToken cancellationToken) in /_/src/SoulseekClient.cs:line 3262
   at Soulseek.SoulseekClient.DownloadToStreamAsync(String username, String remoteFilename, Func`1 outputStreamFactory, Nullable`1 size, Int64 startOffset, Int32 token, TransferOptions options, CancellationToken cancellationToken) in /_/src/SoulseekClient.cs:line 3302
   at Soulseek.SoulseekClient.DownloadToFileAsync(String username, String remoteFilename, String localFilename, Nullable`1 size, Int64 startOffset, Int32 token, TransferOptions options, CancellationToken cancellationToken) in /_/src/SoulseekClient.cs:line 3018
   at Soulseek.Tests.Unit.Client.DownloadAsyncTests.DownloadToFileAsync_Uses_Size_Given_Size_When_Skipping_Queue(String username, IPEndPoint endpoint, String filename, Int32 token, Int32 size, Int64 givenSize) in /root/project/tests/Soulseek.Tests.Unit/Client/DownloadAsyncTests.cs:line 1139
--- End of stack trace from previous location ---
----- Inner Stack Trace -----
   at Soulseek.SoulseekClient.DownloadToStreamAsync(String username, String remoteFilename, Func`1 outputStreamFactory, Nullable`1 size, Int64 startOffset, Int32 token, TransferOptions options, CancellationToken cancellationToken) in /_/src/SoulseekClient.cs:line 3221
[xUnit.net 00:00:04.99]     DownloadToFileAsync raises Download events on failure(username: "usernameec381aae-7368-4dab-aff5-f30ce0029546", endpoint: 127.0.0.1:8, filename: "filenamedfaa1d6e-cfd3-41fd-9507-4282e1d4149b", token: 64, size: 114) [FAIL]
  Failed DownloadToFileAsync raises Download events on failure(username: "usernameec381aae-7368-4dab-aff5-f30ce0029546", endpoint: 127.0.0.1:8, filename: "filenamedfaa1d6e-cfd3-41fd-9507-4282e1d4149b", token: 64, size: 114) [14 ms]
  Error Message:
   Assert.Equal() Failure
Expected: InProgress
Actual:   Initializing
  Stack Trace:
     at Soulseek.Tests.Unit.Client.DownloadAsyncTests.DownloadToFileAsync_Raises_Download_Events_On_Failure(String username, IPEndPoint endpoint, String filename, Int32 token, Int32 size) in /root/project/tests/Soulseek.Tests.Unit/Client/DownloadAsyncTests.cs:line 2444
--- End of stack trace from previous location ---
[xUnit.net 00:00:05.00]     DownloadToFileAsync uses size from TransferResponse given null size when queued(username: "usernamedeb02c2b-3e59-4471-82fa-5e6603f315b8", endpoint: 127.0.0.1:24, filename: "filename05d17f3f-cee3-403b-96e3-66049ce866f5", token: 224, size: 213) [FAIL]
  Failed DownloadToFileAsync uses size from TransferResponse given null size when queued(username: "usernamedeb02c2b-3e59-4471-82fa-5e6603f315b8", endpoint: 127.0.0.1:24, filename: "filename05d17f3f-cee3-403b-96e3-66049ce866f5", token: 224, size: 213) [8 ms]
  Error Message:
   Soulseek.SoulseekClientException : Failed to download file filename05d17f3f-cee3-403b-96e3-66049ce866f5 from user usernamedeb02c2b-3e59-4471-82fa-5e6603f315b8: Object reference not set to an instance of an object.
---- System.NullReferenceException : Object reference not set to an instance of an object.
  Stack Trace:
     at Soulseek.SoulseekClient.DownloadToStreamAsync(String username, String remoteFilename, Func`1 outputStreamFactory, Nullable`1 size, Int64 startOffset, Int32 token, TransferOptions options, CancellationToken cancellationToken) in /_/src/SoulseekClient.cs:line 3262
   at Soulseek.SoulseekClient.DownloadToStreamAsync(String username, String remoteFilename, Func`1 outputStreamFactory, Nullable`1 size, Int64 startOffset, Int32 token, TransferOptions options, CancellationToken cancellationToken) in /_/src/SoulseekClient.cs:line 3302
   at Soulseek.SoulseekClient.DownloadToFileAsync(String username, String remoteFilename, String localFilename, Nullable`1 size, Int64 startOffset, Int32 token, TransferOptions options, CancellationToken cancellationToken) in /_/src/SoulseekClient.cs:line 3018
   at Soulseek.Tests.Unit.Client.DownloadAsyncTests.DownloadToFileAsync_Uses_Size_From_TransferResponse_When_Queued(String username, IPEndPoint endpoint, String filename, Int32 token, Int32 size) in /root/project/tests/Soulseek.Tests.Unit/Client/DownloadAsyncTests.cs:line 1951
--- End of stack trace from previous location ---
----- Inner Stack Trace -----
   at Soulseek.SoulseekClient.DownloadToStreamAsync(String username, String remoteFilename, Func`1 outputStreamFactory, Nullable`1 size, Int64 startOffset, Int32 token, TransferOptions options, CancellationToken cancellationToken) in /_/src/SoulseekClient.cs:line 3221

This is getting old.

jpdillingham avatar Feb 21 '22 23:02 jpdillingham

I thought I documented this somewhere, but I can't find it now so I might as well repeat it. #757 significantly improved these issues.

The root cause was that many tests related to uploads and downloads were referencing the file 'local' which was ending up being created on disk and getting locked, causing any other subsequent test also referencing this 'local' file to fail due to inability to open the file. Prior to #724, these failures bubbled up as null reference exceptions because the streams that failed to be created due to the underlying I/O failures would be null during finalization. After #724 the I/O exceptions began appearing in tests, which led me to the resolution.

Tests still flake once in a while but are no longer really disruptive.

jpdillingham avatar Nov 27 '22 18:11 jpdillingham