Testcontainers do NOT work on kubernetes self hosted runners, but they DO work on VM self hosted runners and locally
I'm running the tests on exactly the same scenario (same image, resources, code...), with a difference: On VM-self-hosted-runners and/or locally, everything works as expected On Kubernetes-self-hosted-runners(https://github.com/actions/actions-runner-controller/tree/master), it doesn't
After some debugging, I assume the problem is here:
import { GenericContainer } from 'testcontainers';
import { RedisHelper } from './utils/redis.helper';
let container;
export let host: string;
export let port: string;
export let dbNumber: string;
export let redisHelper: RedisHelper;
beforeAll(async () => {
console.log('Starting Redis container'); ##THIS IS PROPERLY LOGGED
container = await new GenericContainer('redis').withExposedPorts(6379).start(); ##FAILS HERE
console.log('Redis container started'); ##THIS IS NOT
console.log(container.getHost());
console.log(container.getMappedPort(6379).toString());
redisHelper = new RedisHelper();
await redisHelper.connectToRedis(container.getHost(), container.getMappedPort(6379).toString(), '0');
process.env.REDIS_HOST = container.getHost();
process.env.REDIS_PORT = container.getMappedPort(6379).toString();
});
Expected Behaviour Containers should be able to communicate properly
Actual Behaviour Connection is never established
Testcontainer Logs
2023-09-20T14:26:44.277Z testcontainers [DEBUG] [e15b38c0264b] Demuxing stream... 2023-09-20T14:26:44.278Z testcontainers [DEBUG] [e15b38c0264b] Demuxed stream 2023-09-20T14:26:44.278Z testcontainers [DEBUG] [e15b38c0264b] Fetched container logs 2023-09-20T14:26:44.281Z testcontainers [DEBUG] [e15b38c0264b] Log wait strategy complete 2023-09-20T14:26:44.282Z testcontainers [INFO] [e15b38c0264b] Container is ready 2023-09-20T14:26:44.283Z testcontainers [DEBUG] [e15b38c0264b] Connecting to Reaper (attempt 1) on "localhost:32780"... 2023-09-20T14:26:44.284Z testcontainers [ERROR] Reaper e15b38c0264b7d35def0ce57224ce2fd291d11d351768bc8468fe96cb89f0a15 socket error: Error: connect ECONNREFUSED 127.0.0.1:32780 2023-09-20T14:26:44.285Z testcontainers [ERROR] [e15b38c0264b] Connection to Reaper closed with error 2023-09-20T14:26:45.285Z testcontainers [DEBUG] [e15b38c0264b] Connecting to Reaper (attempt 2) on "localhost:32780"... 2023-09-20T14:26:45.286Z testcontainers [ERROR] Reaper e15b38c0264b7d35def0ce57224ce2fd291d11d351768bc8468fe96cb89f0a15 socket error: Error: connect ECONNREFUSED 127.0.0.1:32780 2023-09-20T14:26:45.286Z testcontainers [ERROR] [e15b38c0264b] Connection to Reaper closed with error 2023-09-20T14:26:46.286Z testcontainers [DEBUG] [e15b38c0264b] Connecting to Reaper (attempt 3) on "localhost:32780"... 2023-09-20T14:26:46.287Z testcontainers [ERROR] Reaper e15b38c0264b7d35def0ce57224ce2fd291d11d351768bc8468fe96cb89f0a15 socket error: Error: connect ECONNREFUSED 127.0.0.1:32780 2023-09-20T14:26:46.287Z testcontainers [ERROR] [e15b38c0264b] Connection to Reaper closed with error 2023-09-20T14:26:47.287Z testcontainers [DEBUG] [e15b38c0264b] Connecting to Reaper (attempt 4) on "localhost:32780"... 2023-09-20T14:26:47.288Z testcontainers [ERROR] Reaper e15b38c0264b7d35def0ce57224ce2fd291d11d351768bc8468fe96cb89f0a15 socket error: Error: connect ECONNREFUSED 127.0.0.1:32780 2023-09-20T14:26:47.288Z testcontainers [ERROR] [e15b38c0264b] Connection to Reaper closed with error 2023-09-20T14:26:48.289Z testcontainers [DEBUG] [e15b38c0264b] Connecting to Reaper (attempt 5) on "localhost:32780"... 2023-09-20T14:26:48.290Z testcontainers [ERROR] Reaper e15b38c0264b7d35def0ce57224ce2fd291d11d351768bc8468fe96cb89f0a15 socket error: Error: connect ECONNREFUSED 127.0.0.1:32780 2023-09-20T14:26:48.290Z testcontainers [ERROR] [e15b38c0264b] Connection to Reaper closed with error 2023-09-20T14:26:48.290Z testcontainers [ERROR] [e15b38c0264b] Failed to connect to Reaper 2023-09-20T14:26:48.290Z testcontainers [DEBUG] Releasing lock file "/tmp/testcontainers-node.lock"... 2023-09-20T14:26:48.290Z testcontainers [DEBUG] Released lock file "/tmp/testcontainers-node.lock"
Steps to Reproduce
- Connect to the runner Pod with kubectl exec -it [POD] /bin/bash
- git clone {repo with tests}
- curl -o- https://raw.githubusercontent.com/nvm-sh/nvm/v0.39.3/install.sh | bash + refresh session (in case your image does not include node/npm/etc)
- cd repo + nvm install 18
- npm install; npm run test — And it FAILS - logs above.
Environment Information
- Operating System: Ubuntu 22
- Docker Version: Docker version 24.0.5, build ced0996
- Node version: v18.18.0
- Testcontainers version: ^10.2.1
Any solutions on this one?
How was Docker installed on the pods? Could you try these environment variables:
TESTCONTAINERS_RYUK_PRIVILEGED=false
If that doesn't work try
TESTCONTAINERS_RYUK_DISABLED=true
Also please share all of the logs, not just the error.
This env variable fix to me this issue.
TESTCONTAINERS_RYUK_DISABLED=true
PS: I am using Colima instead Docker stop for MacosX