apm-agent-dotnet icon indicating copy to clipboard operation
apm-agent-dotnet copied to clipboard

Published 20 hours ago • verified publisher icon elastic

Docker tests are failing

Open gregkalapos opened this issue 1 year ago • 2 comments

Already saw 2 fails right after each other on main and now it also failed in one PR.

Example here.

Log:

Error
System.TypeInitializationException : The type initializer for 'DotNet.Testcontainers.Client.DockerApiClient' threw an exception.
---- Docker.DotNet.DockerApiException : Docker API responded with status code=BadRequest, response=400 Bad Request
Stacktrace
System.TypeInitializationException : The type initializer for 'DotNet.Testcontainers.Client.DockerApiClient' threw an exception.
---- Docker.DotNet.DockerApiException : Docker API responded with status code=BadRequest, response=400 Bad Request
Stack Trace:
   at DotNet.Testcontainers.Client.DockerApiClientImage.ByPropertyAsync(String property, String value)
   at DotNet.Testcontainers.Client.DockerApiClientImage.ByNameAsync(String name)
   at DotNet.Testcontainers.Client.AbstractContainerImageClient`1.ExistsWithNameAsync(String name)
   at DotNet.Testcontainers.Client.TestcontainersClient.RunAsync(TestcontainersConfiguration config, CancellationToken ct)
   at DotNet.Testcontainers.Containers.Modules.TestcontainersContainer.Create()
   at DotNet.Testcontainers.Containers.Modules.TestcontainersContainer.StartAsync()
   at Elastic.Apm.SqlClient.Tests.SqlServerFixture.InitializeAsync() in /var/lib/jenkins/workspace/net_apm-agent-dotnet-mbp_PR-1818/apm-agent-dotnet/test/Elastic.Apm.SqlClient.Tests/SqlServerFixture.cs:line 35
----- Inner Stack Trace -----
   at Docker.DotNet.DockerClient.HandleIfErrorResponseAsync(HttpStatusCode statusCode, HttpResponseMessage response, IEnumerable`1 handlers)
   at Docker.DotNet.DockerClient.MakeRequestAsync(IEnumerable`1 errorHandlers, HttpMethod method, String path, IQueryString queryString, IRequestContent body, IDictionary`2 headers, TimeSpan timeout, CancellationToken token)
   at Docker.DotNet.SystemOperations.GetSystemInfoAsync(CancellationToken cancellationToken)
   at DotNet.Testcontainers.Client.DockerApiClient..cctor()
Standard Output
Data collection : Logging DataCollector Diagnostics in file: /var/lib/jenkins/workspace/net_apm-agent-dotnet-mbp_PR-1818/apm-agent-dotnet/target/diag-ElasticApmAgent.datacollector.22-09-19_16-41-14_45388_5.log
Logging TestHost Diagnostics in file: /var/lib/jenkins/workspace/net_apm-agent-dotnet-mbp_PR-1818/apm-agent-dotnet/target/diag-ElasticApmAgent.host.22-09-19_16-41-17_82360_5.log
[xUnit.net 00:00:00.00] xUnit.net VSTest Adapter v2.4.3+1b45f5407b (64-bit .NET Core 3.1.0)
[xUnit.net 00:00:01.27]   Discovering: Elastic.Apm.SqlClient.Tests
[xUnit.net 00:00:01.33]   Discovered:  Elastic.Apm.SqlClient.Tests
[xUnit.net 00:00:01.33]   Starting:    Elastic.Apm.SqlClient.Tests
[xUnit.net 00:00:01.55]       System.TypeInitializationException : The type initializer for 'DotNet.Testcontainers.Client.DockerApiClient' threw an exception.
[xUnit.net 00:00:01.55]       ---- Docker.DotNet.DockerApiException : Docker API responded with status code=BadRequest, response=400 Bad Request
[xUnit.net 00:00:01.55]       Stack Trace:
[xUnit.net 00:00:01.56]            at DotNet.Testcontainers.Client.DockerApiClientImage.ByPropertyAsync(String property, String value)
[xUnit.net 00:00:01.56]            at DotNet.Testcontainers.Client.DockerApiClientImage.ByNameAsync(String name)
[xUnit.net 00:00:01.56]            at DotNet.Testcontainers.Client.AbstractContainerImageClient`1.ExistsWithNameAsync(String name)
[xUnit.net 00:00:01.56]            at DotNet.Testcontainers.Client.TestcontainersClient.RunAsync(TestcontainersConfiguration config, CancellationToken ct)
[xUnit.net 00:00:01.56]            at DotNet.Testcontainers.Containers.Modules.TestcontainersContainer.Create()
[xUnit.net 00:00:01.56]            at DotNet.Testcontainers.Containers.Modules.TestcontainersContainer.StartAsync()
[xUnit.net 00:00:01.56]         /var/lib/jenkins/workspace/net_apm-agent-dotnet-mbp_PR-1818/apm-agent-dotnet/test/Elastic.Apm.SqlClient.Tests/SqlServerFixture.cs(35,0): at Elastic.Apm.SqlClient.Tests.SqlServerFixture.InitializeAsync()
[xUnit.net 00:00:01.56]         ----- Inner Stack Trace -----
[xUnit.net 00:00:01.56]            at Docker.DotNet.DockerClient.HandleIfErrorResponseAsync(HttpStatusCode statusCode, HttpResponseMessage response, IEnumerable`1 handlers)
[xUnit.net 00:00:01.56]            at Docker.DotNet.DockerClient.MakeRequestAsync(IEnumerable`1 errorHandlers, HttpMethod method, String path, IQueryString queryString, IRequestContent body, IDictionary`2 headers, TimeSpan timeout, CancellationToken token)
[xUnit.net 00:00:01.56]            at Docker.DotNet.SystemOperations.GetSystemInfoAsync(CancellationToken cancellationToken)
[xUnit.net 00:00:01.56]            at DotNet.Testcontainers.Client.DockerApiClient..cctor()
[xUnit.net 00:00:01.61]       System.TypeInitializationException : The type initializer for 'DotNet.Testcontainers.Client.DockerApiClient' threw an exception.
[xUnit.net 00:00:01.61]       ---- Docker.DotNet.DockerApiException : Docker API responded with status code=BadRequest, response=400 Bad Request
[xUnit.net 00:00:01.61]       Stack Trace:
[xUnit.net 00:00:01.61]            at DotNet.Testcontainers.Client.DockerApiClientImage.ByPropertyAsync(String property, String value)
[xUnit.net 00:00:01.61]            at DotNet.Testcontainers.Client.DockerApiClientImage.ByNameAsync(String name)
[xUnit.net 00:00:01.61]            at DotNet.Testcontainers.Client.AbstractContainerImageClient`1.ExistsWithNameAsync(String name)
[xUnit.net 00:00:01.61]            at DotNet.Testcontainers.Client.TestcontainersClient.RunAsync(TestcontainersConfiguration config, CancellationToken ct)
[xUnit.net 00:00:01.61]            at DotNet.Testcontainers.Containers.Modules.TestcontainersContainer.Create()
[xUnit.net 00:00:01.61]            at DotNet.Testcontainers.Containers.Modules.TestcontainersContainer.StartAsync()
[xUnit.net 00:00:01.61]         /var/lib/jenkins/workspace/net_apm-agent-dotnet-mbp_PR-1818/apm-agent-dotnet/test/Elastic.Apm.SqlClient.Tests/SqlServerFixture.cs(35,0): at Elastic.Apm.SqlClient.Tests.SqlServerFixture.InitializeAsync()
[xUnit.net 00:00:01.61]         ----- Inner Stack Trace -----
[xUnit.net 00:00:01.61]            at Docker.DotNet.DockerClient.HandleIfErrorResponseAsync(HttpStatusCode statusCode, HttpResponseMessage response, IEnumerable`1 handlers)
[xUnit.net 00:00:01.61]            at Docker.DotNet.DockerClient.MakeRequestAsync(IEnumerable`1 errorHandlers, HttpMethod method, String path, IQueryString queryString, IRequestContent body, IDictionary`2 headers, TimeSpan timeout, CancellationToken token)
[xUnit.net 00:00:01.61]            at Docker.DotNet.SystemOperations.GetSystemInfoAsync(CancellationToken cancellationToken)
[xUnit.net 00:00:01.62]            at DotNet.Testcontainers.Client.DockerApiClient..cctor()
[xUnit.net 00:00:01.62]       System.TypeInitializationException : The type initializer for 'DotNet.Testcontainers.Client.DockerApiClient' threw an exception.
[xUnit.net 00:00:01.62]       ---- Docker.DotNet.DockerApiException : Docker API responded with status code=BadRequest, response=400 Bad Request
[xUnit.net 00:00:01.62]       Stack Trace:
[xUnit.net 00:00:01.62]            at DotNet.Testcontainers.Client.DockerApiClientImage.ByPropertyAsync(String property, String value)
[xUnit.net 00:00:01.62]            at DotNet.Testcontainers.Client.DockerApiClientImage.ByNameAsync(String name)
[xUnit.net 00:00:01.62]            at DotNet.Testcontainers.Client.AbstractContainerImageClient`1.ExistsWithNameAsync(String name)
[xUnit.net 00:00:01.62]            at DotNet.Testcontainers.Client.TestcontainersClient.RunAsync(TestcontainersConfiguration config, CancellationToken ct)
[xUnit.net 00:00:01.62]            at DotNet.Testcontainers.Containers.Modules.TestcontainersContainer.Create()
[xUnit.net 00:00:01.62]            at DotNet.Testcontainers.Containers.Modules.TestcontainersContainer.StartAsync()
[xUnit.net 00:00:01.62]         /var/lib/jenkins/workspace/net_apm-agent-dotnet-mbp_PR-1818/apm-agent-dotnet/test/Elastic.Apm.SqlClient.Tests/SqlServerFixture.cs(35,0): at Elastic.Apm.SqlClient.Tests.SqlServerFixture.InitializeAsync()
[xUnit.net 00:00:01.62]         ----- Inner Stack Trace -----
[xUnit.net 00:00:01.62]            at Docker.DotNet.DockerClient.HandleIfErrorResponseAsync(HttpStatusCode statusCode, HttpResponseMessage response, IEnumerable`1 handlers)
[xUnit.net 00:00:01.62]            at Docker.DotNet.DockerClient.MakeRequestAsync(IEnumerable`1 errorHandlers, HttpMethod method, String path, IQueryString queryString, IRequestContent body, IDictionary`2 headers, TimeSpan timeout, CancellationToken token)
[xUnit.net 00:00:01.62]            at Docker.DotNet.SystemOperations.GetSystemInfoAsync(CancellationToken cancellationToken)
[xUnit.net 00:00:01.62]            at DotNet.Testcontainers.Client.DockerApiClient..cctor()
[xUnit.net 00:00:01.63]       System.TypeInitializationException : The type initializer for 'DotNet.Testcontainers.Client.DockerApiClient' threw an exception.
[xUnit.net 00:00:01.63]       ---- Docker.DotNet.DockerApiException : Docker API responded with status code=BadRequest, response=400 Bad Request
[xUnit.net 00:00:01.63]       Stack Trace:
[xUnit.net 00:00:01.63]            at DotNet.Testcontainers.Client.DockerApiClientImage.ByPropertyAsync(String property, String value)
[xUnit.net 00:00:01.63]            at DotNet.Testcontainers.Client.DockerApiClientImage.ByNameAsync(String name)
[xUnit.net 00:00:01.63]            at DotNet.Testcontainers.Client.AbstractContainerImageClient`1.ExistsWithNameAsync(String name)
[xUnit.net 00:00:01.63]            at DotNet.Testcontainers.Client.TestcontainersClient.RunAsync(TestcontainersConfiguration config, CancellationToken ct)
[xUnit.net 00:00:01.63]            at DotNet.Testcontainers.Containers.Modules.TestcontainersContainer.Create()
[xUnit.net 00:00:01.63]            at DotNet.Testcontainers.Containers.Modules.TestcontainersContainer.StartAsync()
[xUnit.net 00:00:01.63]         /var/lib/jenkins/workspace/net_apm-agent-dotnet-mbp_PR-1818/apm-agent-dotnet/test/Elastic.Apm.SqlClient.Tests/SqlServerFixture.cs(35,0): at Elastic.Apm.SqlClient.Tests.SqlServerFixture.InitializeAsync()
[xUnit.net 00:00:01.63]         ----- Inner Stack Trace -----
[xUnit.net 00:00:01.63]            at Docker.DotNet.DockerClient.HandleIfErrorResponseAsync(HttpStatusCode statusCode, HttpResponseMessage response, IEnumerable`1 handlers)
[xUnit.net 00:00:01.63]            at Docker.DotNet.DockerClient.MakeRequestAsync(IEnumerable`1 errorHandlers, HttpMethod method, String path, IQueryString queryString, IRequestContent body, IDictionary`2 headers, TimeSpan timeout, CancellationToken token)
[xUnit.net 00:00:01.63]            at Docker.DotNet.SystemOperations.GetSystemInfoAsync(CancellationToken cancellationToken)
[xUnit.net 00:00:01.63]            at DotNet.Testcontainers.Client.DockerApiClient..cctor()
[xUnit.net 00:00:01.64]       System.TypeInitializationException : The type initializer for 'DotNet.Testcontainers.Client.DockerApiClient' threw an exception.
[xUnit.net 00:00:01.64]       ---- Docker.DotNet.DockerApiException : Docker API responded with status code=BadRequest, response=400 Bad Request
[xUnit.net 00:00:01.64]       Stack Trace:
[xUnit.net 00:00:01.64]            at DotNet.Testcontainers.Client.DockerApiClientImage.ByPropertyAsync(String property, String value)
[xUnit.net 00:00:01.64]            at DotNet.Testcontainers.Client.DockerApiClientImage.ByNameAsync(String name)
[xUnit.net 00:00:01.64]            at DotNet.Testcontainers.Client.AbstractContainerImageClient`1.ExistsWithNameAsync(String name)
[xUnit.net 00:00:01.64]            at DotNet.Testcontainers.Client.TestcontainersClient.RunAsync(TestcontainersConfiguration config, CancellationToken ct)
[xUnit.net 00:00:01.64]            at DotNet.Testcontainers.Containers.Modules.TestcontainersContainer.Create()
[xUnit.net 00:00:01.64]            at DotNet.Testcontainers.Containers.Modules.TestcontainersContainer.StartAsync()
[xUnit.net 00:00:01.64]         /var/lib/jenkins/workspace/net_apm-agent-dotnet-mbp_PR-1818/apm-agent-dotnet/test/Elastic.Apm.SqlClient.Tests/SqlServerFixture.cs(35,0): at Elastic.Apm.SqlClient.Tests.SqlServerFixture.InitializeAsync()
[xUnit.net 00:00:01.64]         ----- Inner Stack Trace -----
[xUnit.net 00:00:01.64]            at Docker.DotNet.DockerClient.HandleIfErrorResponseAsync(HttpStatusCode statusCode, HttpResponseMessage response, IEnumerable`1 handlers)
[xUnit.net 00:00:01.64]            at Docker.DotNet.DockerClient.MakeRequestAsync(IEnumerable`1 errorHandlers, HttpMethod method, String path, IQueryString queryString, IRequestContent body, IDictionary`2 headers, TimeSpan timeout, CancellationToken token)
[xUnit.net 00:00:01.64]            at Docker.DotNet.SystemOperations.GetSystemInfoAsync(CancellationToken cancellationToken)
[xUnit.net 00:00:01.64]            at DotNet.Testcontainers.Client.DockerApiClient..cctor()
[xUnit.net 00:00:01.64]   Finished:    Elastic.Apm.SqlClient.Tests
Standard Error
[xUnit.net 00:00:01.55]     Elastic.Apm.SqlClient.Tests.EfCoreWithMsSqlTests.BothEfCoreAndSqlClientCapturingActive [FAIL]
[xUnit.net 00:00:01.61]     Elastic.Apm.SqlClient.Tests.SqlClientListenerTests.SqlClientDiagnosticListener_ShouldCaptureSpan(providerName: "System.Data.SqlClient", connectionCreator: Func`2 { Method = System.Data.Common.DbConnection <get_Connections>b__8_0(System.String), Target = <>c { } }) [FAIL]
[xUnit.net 00:00:01.62]     Elastic.Apm.SqlClient.Tests.SqlClientListenerTests.SqlClientDiagnosticListener_ShouldCaptureSpan(providerName: "Microsoft.Data.SqlClient", connectionCreator: Func`2 { Method = System.Data.Common.DbConnection <get_Connections>b__8_1(System.String), Target = <>c { } }) [FAIL]
[xUnit.net 00:00:01.63]     Elastic.Apm.SqlClient.Tests.SqlClientListenerTests.SqlClientDiagnosticListener_ShouldCaptureErrorFromSystemSqlClient(providerName: "System.Data.SqlClient", connectionCreator: Func`2 { Method = System.Data.Common.DbConnection <get_Connections>b__8_0(System.String), Target = <>c { } }) [FAIL]
[xUnit.net 00:00:01.64]     Elastic.Apm.SqlClient.Tests.SqlClientListenerTests.SqlClientDiagnosticListener_ShouldCaptureErrorFromSystemSqlClient(providerName: "Microsoft.Data.SqlClient", connectionCreator: Func`2 { Method = System.Data.Common.DbConnection <get_Connections>b__8_1(System.String), Target = <>c { } }) [FAIL]
Data collector 'Blame' message: All tests finished running, Sequence file will not be generated.

gregkalapos avatar Sep 19 '22 17:09 gregkalapos

Currently we were using DotNet.Testcontainers version 1.0.0. I updated to 1.6.0 which gave a more useful error message (see below). The root cause seems to be that the docker image we rely on for the MSSql tests is not available.

Docker.DotNet.DockerImageNotFoundException: Docker API responded with status code=NotFound, response={"message":"No such image: ghcr.io/psanetra/ryuk:...

Docker.DotNet.DockerImageNotFoundException
Docker API responded with status code=NotFound, response={"message":"No such image: ghcr.io/psanetra/ryuk:2021.12.20"}

   at Docker.DotNet.ContainerOperations.<>c.<.cctor>b__30_1(HttpStatusCode statusCode, String responseBody)
   at Docker.DotNet.DockerClient.HandleIfErrorResponseAsync(HttpStatusCode statusCode, HttpResponseMessage response, IEnumerable`1 handlers)
   at Docker.DotNet.DockerClient.MakeRequestAsync(IEnumerable`1 errorHandlers, HttpMethod method, String path, IQueryString queryString, IRequestContent body, IDictionary`2 headers, TimeSpan timeout, CancellationToken token)
   at Docker.DotNet.ContainerOperations.CreateContainerAsync(CreateContainerParameters parameters, CancellationToken cancellationToken)
   at DotNet.Testcontainers.Clients.DockerContainerOperations.RunAsync(ITestcontainersConfiguration configuration, CancellationToken ct)
   at DotNet.Testcontainers.Clients.TestcontainersClient.RunAsync(ITestcontainersConfiguration configuration, CancellationToken ct)
   at DotNet.Testcontainers.Containers.TestcontainersContainer.Create(CancellationToken ct)
   at DotNet.Testcontainers.Containers.TestcontainersContainer.StartAsync(CancellationToken ct)
   at DotNet.Testcontainers.Containers.ResourceReaper.GetAndStartNewAsync(Guid sessionId, IDockerEndpointAuthenticationConfiguration dockerEndpointAuthConfig, String ryukImage, TimeSpan initTimeout, CancellationToken ct)
   at DotNet.Testcontainers.Containers.ResourceReaper.GetAndStartNewAsync(Guid sessionId, IDockerEndpointAuthenticationConfiguration dockerEndpointAuthConfig, String ryukImage, TimeSpan initTimeout, CancellationToken ct)
   at DotNet.Testcontainers.Containers.ResourceReaper.GetAndStartDefaultAsync(IDockerEndpointAuthenticationConfiguration dockerEndpointAuthConfig, CancellationToken ct)
   at DotNet.Testcontainers.Clients.TestcontainersClient.RunAsync(ITestcontainersConfiguration configuration, CancellationToken ct)
   at DotNet.Testcontainers.Containers.TestcontainersContainer.Create(CancellationToken ct)
   at DotNet.Testcontainers.Containers.TestcontainersContainer.StartAsync(CancellationToken ct)
   at Elastic.Apm.SqlClient.Tests.SqlServerFixture.InitializeAsync() in /Users/gregkalapos/repos/gregkalapos_apm-agent-dotnet/test/Elastic.Apm.SqlClient.Tests/SqlServerFixture.cs:line 33
   at Xunit.Sdk.ExceptionAggregator.RunAsync(Func`1 code) in C:\Dev\xunit\xunit\src\xunit.core\Sdk\ExceptionAggregator.cs:line 90



-----

System.InvalidOperationException
Testcontainer has not been created.
   at DotNet.Testcontainers.Containers.TestcontainersContainer.ThrowIfContainerHasNotBeenCreated()
   at DotNet.Testcontainers.Containers.TestcontainersContainer.get_Id()
   at DotNet.Testcontainers.Containers.TestcontainersContainer.StopAsync(CancellationToken ct)
   at Elastic.Apm.SqlClient.Tests.SqlServerFixture.DisposeAsync() in /Users/gregkalapos/repos/gregkalapos_apm-agent-dotnet/test/Elastic.Apm.SqlClient.Tests/SqlServerFixture.cs:line 37
   at Xunit.Sdk.ExceptionAggregator.RunAsync(Func`1 code) in C:\Dev\xunit\xunit\src\xunit.core\Sdk\ExceptionAggregator.cs:line 90

gregkalapos avatar Sep 20 '22 08:09 gregkalapos

The elasticsearch docker test is still faling: https://apm-ci.elastic.co/job/apm-agent-dotnet/job/apm-agent-dotnet-mbp/job/PR-1826/2/testReport/Elastic.Apm.Elasticsearch.Tests/ElasticsearchTests/Initializing___Parallel___Windows__NET_Core___Test___ElasticsearchTests_Elastic_Apm_Elasticsearch_Tests_ElasticsearchTests_Elasticsearch_Span_Does_Not_Have_Http_Child_Span/

Error
[Test Class Cleanup Failure (Elastic.Apm.Elasticsearch.Tests.ElasticsearchTests)]: System.InvalidOperationException : Testcontainer has not been created.
Stacktrace
[Test Class Cleanup Failure (Elastic.Apm.Elasticsearch.Tests.ElasticsearchTests)]: System.InvalidOperationException : Testcontainer has not been created.
Stack Trace:
   at DotNet.Testcontainers.Containers.TestcontainersContainer.ThrowIfContainerHasNotBeenCreated()
   at DotNet.Testcontainers.Containers.TestcontainersContainer.get_Id()
   at DotNet.Testcontainers.Containers.TestcontainersContainer.StopAsync(CancellationToken ct)
   at Elastic.Apm.Elasticsearch.Tests.ElasticsearchFixture.DisposeAsync() in C:\Users\jenkins\workspace\net_apm-agent-dotnet-mbp_PR-1826\apm-agent-dotnet\test\Elastic.Apm.Elasticsearch.Tests\ElasticsearchFixture.cs:line 35

gregkalapos avatar Sep 23 '22 10:09 gregkalapos

https://github.com/elastic/apm-agent-dotnet/pull/1834 addresses the issue above.

Since builds were randomly failing and not all builds failed, I leave this issue open for a few days.

gregkalapos avatar Sep 26 '22 21:09 gregkalapos

Seems to be fixed now.

gregkalapos avatar Oct 20 '22 13:10 gregkalapos