[Bug]: Sporadically fails to create reaper in GitHub Actions

[abemedia]

Testcontainers version

v0.27.0

Using the latest Testcontainers version?

Yes

Host OS

Ubuntu 22.04.3 LTS

Host arch

amd64

Go version

1.20

Docker version

GitHub Actions 🤷🏼‍♂️

Docker info

GitHub Actions 🤷🏼‍♂️

What happened?

This does not seem to happen when running locally but occasionally my tests fail in GitHub Actions with the error creating reaper failed.

Relevant log output

2024/01/28 17:56:02 github.com/testcontainers/testcontainers-go - Connected to docker: 
  Server Version: 24.0.7
  API Version: 1.43
  Operating System: Ubuntu 22.04.3 LTS
  Total Memory: 6921 MB
  Resolved Docker Host: unix:///var/run/docker.sock
  Resolved Docker Socket Path: /var/run/docker.sock
  Test SessionID: 1db7667978fa78cee3873c39e54eaa6e9e4e8bd9b8a5025ceb0b2b7c20578725
  Test ProcessID: c0f2e63c-5ad4-4ae2-a60d-de0250ff089f
2024/01/28 17:56:02 🔥 Reaper obtained from Docker for this test session 24e370b447d41033419fea8aefd579d45338a16882557d81a9ee8e0df38cfaad
2024/01/28 17:56:02 port not found: creating reaper failed: failed to create container

Additional information

No response

[Bablzz]

Hi @abemedia ! I hope you are doing well! can you share link to the job? I check actions in this repo - repear works fine - for example - https://github.com/testcontainers/testcontainers-go/actions/runs/7730045562/job/21075148438#step:10:41

[kostyay]

Did you manage to solve this issue? I'm facing an issue with reaper failing with the following error.. it only happens on GHA:

2024/02/13 14:47:24 github.com/testcontainers/testcontainers-go - Connected to docker: 
  Server Version: 25.0.3
  API Version: 1.43
  Operating System: Alpine Linux v3.19 (containerized)
  Total Memory: 31291 MB
  Resolved Docker Host: unix:///run/docker.sock
  Resolved Docker Socket Path: /run/docker.sock
  Test SessionID: c00c76c83c7180c78797a94d2146b5216f91200d033f9eb87efd620b53b27115
  Test ProcessID: e60189dd-ddfc-4b4c-98ab-1[70](xxx)99ea7a937
2024/02/13 14:47:24 🐳 Creating container for image testcontainers/ryuk:0.6.0
    internal_handler_test.go:283: failed to start container: look up reaper container returned nil although creation failed due to name conflict: creating reaper failed: failed to create container

[pablocalvo-zh]

Did you manage to solve this issue? I'm facing an issue with reaper failing with the following error.. it only happens on GHA:

2024/02/13 14:47:24 github.com/testcontainers/testcontainers-go - Connected to docker: 
  Server Version: 25.0.3
  API Version: 1.43
  Operating System: Alpine Linux v3.19 (containerized)
  Total Memory: 31291 MB
  Resolved Docker Host: unix:///run/docker.sock
  Resolved Docker Socket Path: /run/docker.sock
  Test SessionID: c00c76c83c7180c78797a94d2146b5216f91200d033f9eb87efd620b53b27115
  Test ProcessID: e60189dd-ddfc-4b4c-98ab-1[70](xxx)99ea7a937
2024/02/13 14:47:24 🐳 Creating container for image testcontainers/ryuk:0.6.0
    internal_handler_test.go:283: failed to start container: look up reaper container returned nil although creation failed due to name conflict: creating reaper failed: failed to create container

I am getting the same error apparently.

[pablocalvo-zh]

@kostyay I was able to bypass this error by cleaning up containers on the tests and then using TESTCONTAINERS_RYUK_DISABLED: true. FYI I realize this happens mostly when there are multiple tests spawning containers.

https://java.testcontainers.org/features/configuration/

[webstradev]

I am experiencing this exact same issue in gitlab runners when running a few different jobs that are all using testcontainers. 4/5 of my jobs always pass and one job (which runs tests in parallel) always fails when the other 4 are running but passes when I retry it (and it is running by itself).

[abdulkk49]

I am getting a similar error in Github Actions:

Server Version: 25.0.2
  API Version: 1.44
  Operating System: Ubuntu 20.04.6 LTS
  Total Memory: 7751 MB
  Resolved Docker Host: unix:///var/run/docker.sock
  Resolved Docker Socket Path: /var/run/docker.sock
2024/03/19 13:49:59 🐳 Creating container for image testcontainers/ryuk:0.6.0
        	Error:      	Received unexpected error:
        	            	look up reaper container returned nil although creation failed due to name conflict: creating reaper failed: failed to create container

[shepherdjerred]

I've seen this quite frequently in Jenkins when running a large number of tests. Disabling the reaper with TESTCONTAINERS_RYUK_DISABLED fixed the issue for me.

If you're in an ephemeral environment enabling TESTCONTAINERS_RYUK_DISABLED should be a fine workaround without any negative consequences.

[tuxxi]

Adding a data point here: we never hit this until upgrading testcontainers-go from 0.19 to 0.28.
Soon after, starting hitting this issue fairly regularly (I would estimate about 1 out of 4 jobs fail due to either

look up reaper container returned nil although creation failed due to name conflict

or

connecting to reaper failed: failed to create container

[bhlox]

Still having this issue. I notice this when multiple containers are spawned. But when done individually, no errors of reaper or port issues occurred. Having the (cached) text makes the test pass.

--- PASS: TestProducts (3.18s)
    --- PASS: TestProducts/getting_product_of_23 (0.00s)
    --- PASS: TestProducts/getting_product_of_14 (0.00s)
    --- PASS: TestProducts/getting_product_of_4fh555sd (0.00s)
    --- PASS: TestProducts/getting_product_of_-23.42 (0.00s)
    --- PASS: TestProducts/getting_product_of_234234234 (0.00s)
    --- PASS: TestProducts/creating_product_case_index:_0 (0.36s)
    --- PASS: TestProducts/creating_product_case_index:_1 (0.25s)
    --- PASS: TestProducts/creating_product_case_index:_2 (0.25s)
PASS
ok      github.com/bhlox/ecom/internal/services/product (cached)

[kdescoteaux-uptycs]

To narrow @tuxxi 's anecdote. I didn't see this until we upgraded from 0.26 to 0.27 which indirectly changes the ryuk image version as well. Downgrading back is not an option because we did it to be able to use TESTCONTAINERS_HUB_IMAGE_NAME_PREFIX