Postgres on CI/CD won't run

[Wasenshi123]

I use the package to run the test with PostgresDB. I followed your example here and it works fine, locally.

But when I run it on the ci/cd, it won't work.

Basically, it says :

System.Net.Http.HttpRequestException : Connection failed
---- System.Net.Internals.SocketExceptionFactory+ExtendedSocketException : Cannot assign requested address /var/run/docker.sock

[Deffiss]

Hi @Wasenshi123 , it looks like your CI/CD builder agent doesn't have docker installed on it. Could you check this somehow, please?

[Wasenshi123]

Hi @Deffiss , yes, I checked my CI/CD and it indeed doesn't have docker installed in the build process.

So I went on and installed docker but now I got the new error :

Error Message:
   System.Net.Http.HttpRequestException : The requested failed, see inner exception for details.
---- System.IO.IOException : Unexpected end of stream
  Stack Trace:
     at Microsoft.Net.Http.Client.HttpConnection.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at Microsoft.Net.Http.Client.ManagedHandler.ProcessRequestAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at Microsoft.Net.Http.Client.ManagedHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.HttpClient.FinishSendAsyncBuffered(Task`1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts)
   at Docker.DotNet.DockerClient.PrivateMakeRequestAsync(TimeSpan timeout, HttpCompletionOption completionOption, HttpMethod method, String path, IQueryString queryString, IDictionary`2 headers, IRequestContent data, CancellationToken cancellationToken)
   at Docker.DotNet.DockerClient.PrivateMakeRequestAsync(TimeSpan timeout, HttpCompletionOption completionOption, HttpMethod method, String path, IQueryString queryString, IDictionary`2 headers, IRequestContent data, CancellationToken cancellationToken)
   at Docker.DotNet.DockerClient.MakeRequestAsync(IEnumerable`1 errorHandlers, HttpMethod method, String path, IQueryString queryString, IRequestContent body, IDictionary`2 headers, TimeSpan timeout, CancellationToken token)
   at Docker.DotNet.ImageOperations.ListImagesAsync(ImagesListParameters parameters, CancellationToken cancellationToken)
   at TestEnvironment.Docker.DockerEnvironment.PullRequiredImages(CancellationToken token)
   at TestEnvironment.Docker.DockerEnvironment.Up(CancellationToken token)
   at Wasenshi.HemoDialysisPro.Test.Fixture.EnvironmentFixture.InitializeAsync() in /root/project/HemoDialysisPro/Wasenshi.HemoDialysisPro.Test/Fixture/EnvironmentFixture.cs:line 47
----- Inner Stack Trace -----
   at Microsoft.Net.Http.Client.BufferedReadStream.EnsureBufferedAsync(CancellationToken cancel)
   at Microsoft.Net.Http.Client.BufferedReadStream.ReadLineAsync(CancellationToken cancel)
   at Microsoft.Net.Http.Client.HttpConnection.ReadResponseLinesAsync(CancellationToken cancellationToken)
   at Microsoft.Net.Http.Client.HttpConnection.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)

Can you help me out, please? Here is a part of my CI/CD setup in case it helps (circleci config) :

test:
    docker:
      - image: mcr.microsoft.com/dotnet/core/sdk:3.1
    steps:
      - checkout
      - setup_remote_docker
      - run:
          name: Install Docker
          command: |
            apt update
            apt install apt-transport-https ca-certificates curl gnupg2 software-properties-common -y
            curl -fsSL https://download.docker.com/linux/debian/gpg | apt-key add -
            add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/debian $(lsb_release -cs) stable"
            apt update
            apt-cache policy docker-ce
            apt install docker-ce -y
      - run:
          name: Install Docker Compose
          command: |
            set -x
            curl -L https://github.com/docker/compose/releases/download/1.25.3/docker-compose-`uname -s`-`uname -m` > /usr/local/bin/docker-compose
            chmod +x /usr/local/bin/docker-compose
      - run:
          name: Test
          command: dotnet test HemoDialysisPro/HemoDialysisPro.sln -c Release

[Hellevar]

Error Message:
   System.Net.Http.HttpRequestException : The requested failed, see inner exception for details.
---- System.IO.IOException : Unexpected end of stream
  Stack Trace:
     at Microsoft.Net.Http.Client.HttpConnection.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at Microsoft.Net.Http.Client.ManagedHandler.ProcessRequestAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at Microsoft.Net.Http.Client.ManagedHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.HttpClient.FinishSendAsyncBuffered(Task`1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts)
   at Docker.DotNet.DockerClient.PrivateMakeRequestAsync(TimeSpan timeout, HttpCompletionOption completionOption, HttpMethod method, String path, IQueryString queryString, IDictionary`2 headers, IRequestContent data, CancellationToken cancellationToken)
   at Docker.DotNet.DockerClient.PrivateMakeRequestAsync(TimeSpan timeout, HttpCompletionOption completionOption, HttpMethod method, String path, IQueryString queryString, IDictionary`2 headers, IRequestContent data, CancellationToken cancellationToken)
   at Docker.DotNet.DockerClient.MakeRequestAsync(IEnumerable`1 errorHandlers, HttpMethod method, String path, IQueryString queryString, IRequestContent body, IDictionary`2 headers, TimeSpan timeout, CancellationToken token)
   at Docker.DotNet.ImageOperations.ListImagesAsync(ImagesListParameters parameters, CancellationToken cancellationToken)
   at TestEnvironment.Docker.DockerEnvironment.PullRequiredImages(CancellationToken token)
   at TestEnvironment.Docker.DockerEnvironment.Up(CancellationToken token)
   at Wasenshi.HemoDialysisPro.Test.Fixture.EnvironmentFixture.InitializeAsync() in /root/project/HemoDialysisPro/Wasenshi.HemoDialysisPro.Test/Fixture/EnvironmentFixture.cs:line 47
----- Inner Stack Trace -----
   at Microsoft.Net.Http.Client.BufferedReadStream.EnsureBufferedAsync(CancellationToken cancel)
   at Microsoft.Net.Http.Client.BufferedReadStream.ReadLineAsync(CancellationToken cancel)
   at Microsoft.Net.Http.Client.HttpConnection.ReadResponseLinesAsync(CancellationToken cancellationToken)
   at Microsoft.Net.Http.Client.HttpConnection.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)

Seem's like Docker is not able to download requested image from Hub. If you are specifying private docker hub or with access control(login/password or certificate) - it is currently impossible to use it with our lib in the such way

[Wasenshi123]

Hi @Hellevar , No, I didn't use any private image. Please note that this error is from when the method DockerEnvironment.Up() is called.

Another thing I noticed is, the number of total tests is messed up and seems off. I have only 13 tests at a time, but it shows that I have 26? (it runs twice? why?) when I try using dotnet command line to run test manually on the local machine, I got another number, 20

Also another minor note is that it also yields this error at around the end something like:

Error Message:
   [Test Class Cleanup Failure (Wasenshi.HemoDialysisPro.Test.HealthCheckTest)]: System.NullReferenceException : Object reference not set to an instance of an object.
  Stack Trace:
     at Wasenshi.HemoDialysisPro.Test.Fixture.EnvironmentFixture.DisposeAsync() in /root/project/HemoDialysisPro/Wasenshi.HemoDialysisPro.Test/Fixture/EnvironmentFixture.cs:line 33
  X Wasenshi.HemoDialysisPro.Test.UsersControllerTest.Post_EditRoleReturnsFailToRegularUser [1ms]
  Error Message:
   [Test Class Cleanup Failure (Wasenshi.HemoDialysisPro.Test.UsersControllerTest)]: System.NullReferenceException : Object reference not set to an instance of an object.
  Stack Trace:
     at Wasenshi.HemoDialysisPro.Test.Fixture.EnvironmentFixture.DisposeAsync() in /root/project/HemoDialysisPro/Wasenshi.HemoDialysisPro.Test/Fixture/EnvironmentFixture.cs:line 33
  X Wasenshi.HemoDialysisPro.Test.UsersControllerTest.Post_RegisterReturnsFailToRegularUser [1ms]
  Error Message:
   [Test Class Cleanup Failure (Wasenshi.HemoDialysisPro.Test.UsersControllerTest)]: System.NullReferenceException : Object reference not set to an instance of an object.
  Stack Trace:
     at Wasenshi.HemoDialysisPro.Test.Fixture.EnvironmentFixture.DisposeAsync() in /root/project/HemoDialysisPro/Wasenshi.HemoDialysisPro.Test/Fixture/EnvironmentFixture.cs:line 33
  X Wasenshi.HemoDialysisPro.Test.UsersControllerTest.Post_EditUserReturnsFailForDifferentUser [1ms]
  Error Message:
   [Test Class Cleanup Failure (Wasenshi.HemoDialysisPro.Test.UsersControllerTest)]: System.NullReferenceException : Object reference not set to an instance of an object.
  Stack Trace:
     at Wasenshi.HemoDialysisPro.Test.Fixture.EnvironmentFixture.DisposeAsync() in /root/project/HemoDialysisPro/Wasenshi.HemoDialysisPro.Test/Fixture/EnvironmentFixture.cs:line 33
  X Wasenshi.HemoDialysisPro.Test.UsersControllerTest.Post_EditUserReturnsSuccessForOwnerUser [1ms]
  Error Message:
   [Test Class Cleanup Failure (Wasenshi.HemoDialysisPro.Test.UsersControllerTest)]: System.NullReferenceException : Object reference not set to an instance of an object.
  Stack Trace:
     at Wasenshi.HemoDialysisPro.Test.Fixture.EnvironmentFixture.DisposeAsync() in /root/project/HemoDialysisPro/Wasenshi.HemoDialysisPro.Test/Fixture/EnvironmentFixture.cs:line 33

[Update: I try change dotnet test to run debug instead of release on local and it works fine.]

So not sure why but it seems almost like the postgres db docker container cannot build and up fast enough so the test runs before the docker is ready somehow and failed? (To be precise, running test on debug mode for the first time also fail and yields the same error as the release mode, but then next time it can reuse the container and pass successfully)

The error it yields is different from the CI/CD case here, though. Not sure if this is related to each other since I use release mode on CI/CD as per you suggestion

This is the error from running test on release mode (and also for debug mode only on first time) locally :

 X Wasenshi.HemoDialysisPro.Test.HealthCheckTest.Get_HealthCheckSuccess [1ms]
  Error Message:
   Docker.DotNet.DockerApiException : Docker API responded with status code=Conflict, response={"message":"Conflict. The container name \"/Default-postgres\" is already in use by container \"e2ae0cb30bc5ce22ca059c244978e38bb4c14888bd0afedd253c2ef561f87949\". You have to remove (or rename) that container to be able to reuse that name."}

  Stack Trace:
     at Docker.DotNet.DockerClient.HandleIfErrorResponseAsync(HttpStatusCode statusCode, HttpResponseMessage response, IEnumerable`1 handlers)
   at Docker.DotNet.DockerClient.MakeRequestAsync(IEnumerable`1 errorHandlers, HttpMethod method, String path, IQueryString queryString, IRequestContent body, IDictionary`2 headers, TimeSpan timeout, CancellationToken token)
   at Docker.DotNet.ContainerOperations.CreateContainerAsync(CreateContainerParameters parameters, CancellationToken cancellationToken)
   at TestEnvironment.Docker.Container.CreateContainer(String[] environmentVariables, CancellationToken token)
   at TestEnvironment.Docker.Container.RunContainerSafely(String[] environmentVariables, CancellationToken token)
   at TestEnvironment.Docker.Container.Run(IDictionary`2 environmentVariables, CancellationToken token)
   at TestEnvironment.Docker.DockerEnvironment.Up(CancellationToken token)
   at Wasenshi.HemoDialysisPro.Test.Fixture.EnvironmentFixture.InitializeAsync() in D:\Repositories\HemodialysisPro\Hemodialysis-Pro-Backend\HemoDialysisPro\Wasenshi.HemoDialysisPro.Test\Fixture\EnvironmentFixture.cs:line 47

It says this despite that it was the one who is trying to create the container itself (And I've verified that there is none of the container up and running before I run the test)

[Wasenshi123]

Additional information: this is my implementation of the EnvironmentFixture follow your sample

public class EnvironmentFixture : IAsyncLifetime
    {
        private readonly string _environmentName;
        private DockerEnvironment _environment;


        public HttpClient TestClient { get; private set; }

        public EnvironmentFixture()
        {
            _environmentName = "Default";
        }

        public async Task DisposeAsync()
        {
            TestClient.Dispose();
#if !DEBUG
            await _environment.DisposeAsync();
#endif
            await Task.CompletedTask;
        }

        public async Task InitializeAsync()
        {
            // Docker environment seutup.
            _environment = CreateTestEnvironmentBuilder().Build();
            await _environment.Up();

            // API Test host setup
            var postgres = _environment.GetContainer<PostgresContainer>("postgres");
            TestClient = CreateTestClient(postgres);

            await OnInitialized(postgres);
        }

        protected virtual Task OnInitialized(PostgresContainer postgres) => Task.CompletedTask;

        private IDockerEnvironmentBuilder CreateTestEnvironmentBuilder()
        {
            return new DockerEnvironmentBuilder()
                .SetName(_environmentName)
#if DEBUG
                .AddPostgresContainer("postgres", reuseContainer: true);
#else
                .AddPostgresContainer("postgres");
#endif
        }


        private HttpClient CreateTestClient(PostgresContainer postgres)
        {
            var connStr = postgres.GetConnectionString() + $";Database={Guid.NewGuid()}";
            WebApplicationFactory<Startup> factory = new WebApplicationFactory<Startup>();

            return factory.WithWebHostBuilder(x =>
            {
                x.ConfigureAppConfiguration((context, config) =>
                config.AddInMemoryCollection(new[] { new KeyValuePair<string, string>("ConnectionStrings:HemodialysisConnection", connStr) }));
                x.ConfigureServices((context, services) =>
                {
                    services.ConfigureMockJwt();
                });
            }).CreateClient();
        }
    }

[Deffiss]

Did you try to remove all containers: docker rm -f (docker ps -aq)

[Wasenshi123]

Yes, I run that before running dotnet test and still the same outcome.

[Deffiss]

Can you try to remove postgress image (docker rmi) from builder machine and then pull it manually (docker pull)?

[Wasenshi123]

@Deffiss hi,

I've just solved the local problem and now everything works fine just as it should be. (basically, I used the common same name for any Postgres container created by EnvironmentFixture. While in reality, it got created for every test classes, so multiple containers were fighting each other for the name, ha)

But still, the CI/CD remains the same (and that is the main issue). I have no clue about this:

Error Message:
   System.Net.Http.HttpRequestException : The requested failed, see inner exception for details.
---- System.IO.IOException : Unexpected end of stream

FYI: This is running on CircleCi, so it's docker in docker.

[Deffiss]

For docker in docker scenario, you need to configure test env:

return new DockerEnvironmentBuilder()
    ..DockerInDocker(true)

In order to run this from both local and CI/CD envs, you probably need some trick that allows to identify whether you are docker-in-docker or not. We did that from environment variables (something like DOCKER_IN_DOCKER equals TRUE).

[Wasenshi123]

@Deffiss Thank you for your suggestion.

Well, but I have tried it. With .DockerInDocker(true) and also pull Postgres image beforehand, still no luck. I think it has something to do with the circleCi itself.

They have this line - Setup_Remote_Docker that you need to put to be able to run any docker command in the circleCi docker build-agent.

Maybe that raise some security requirements that causes Docker.Dotnet to throw the error above?

(also, still, wondering why it seems to run the test twice in a number of total tests? perhaps it will automatically be gone if the problem above got solved? well, at least in the local it did go away)

[Deffiss]

By default TestEnvironment tries to connect to unix:///var/run/docker.sock if it runs on Linux and to npipe://./pipe/docker_engine if it is on Windows. Maybe for Circile CI you need to specify exact address of remote docker daemon (Setup_Remote_Docker probably means that docker daemon is on the remote machine). You may try to check DOCKER_HOST variable and use its value to pass as an address.

[Deffiss]

I found in this documentation https://circleci.com/docs/2.0/building-docker-images/ that Circle CI creates alias remote-docker so you can try:

            var client = new DockerClientConfiguration(new Uri("http://remote-docker")).CreateClient(); // maybe port should be specified
            var builder = new DockerEnvironmentBuilder(client);