GenericContainer broken after upgrade from 10.9 to 10.10

[jonesbusy]

Hi,

I've facing an issue after upgrade testcontainer node from 10.9 to 10.10 which any other changes

Expected Behaviour

Same as 10.9.0, it should work

Actual Behaviour

Doesn't work, tested with different framework (Backstage and one JS vite application). It's the same

Testcontainer Logs

[DEBUG] Testing container runtime strategy "TestcontainersHostStrategy"...
[DEBUG] Found custom configuration: dockerHost: "tcp://localhost:2475"
[DEBUG] Testing container runtime strategy "ConfigurationStrategy"...
[DEBUG] Container runtime strategy "ConfigurationStrategy" does not work
[DEBUG] Testing container runtime strategy "UnixSocketStrategy"...
[DEBUG] Testing container runtime strategy "RootlessUnixSocketStrategy"...
[DEBUG] Testing container runtime strategy "NpipeSocketStrategy"...

This line is different from

[DEBUG] Container runtime strategy "ConfigurationStrategy" does not work

On 10.9 is

[DEBUG] Container runtime strategy "ConfigurationStrategy" works

Steps to Reproduce

On code

    wireMockContainer = await new GenericContainer("wiremock/wiremock:3.5.4-1")
      .withExposedPorts(8080)
      .withCommand(["--verbose"])
      .withWaitStrategy(Wait.forLogMessage(/port:.*8080/))
      .withStartupTimeout(120000)
      .start();

Environment Information

• Running node 20 LTS
• I'm running testcontainer with kubedock and following env set

DOCKER_HOST = 'tcp://localhost:2475'
TESTCONTAINERS_HUB_IMAGE_NAME_PREFIX  = '*********-docker-io.***************.ch/'

I can provide more details if needed

Thanks

[cristianrgreco]

Hi @jonesbusy,

I gotta say I'm stumped by this one 😄

These are the changes between 10.9 and 10.10 (https://github.com/testcontainers/testcontainers-node/releases/tag/v10.10.0):

🚀 Features
- Add ollama module @ilopezluna (https://github.com/testcontainers/testcontainers-node/pull/778)
- Add RabbitMQ module @gundersong (https://github.com/testcontainers/testcontainers-node/pull/770)

🐛 Bug Fixes
- GenericContainerBuilder false pull policy should not pull image @schummar (https://github.com/testcontainers/testcontainers-node/pull/771)

🧹 Maintenance
- Update minor deps @cristianrgreco (https://github.com/testcontainers/testcontainers-node/pull/780)
- Use new sshd:1.2.0 image @mdelapenya (https://github.com/testcontainers/testcontainers-node/pull/758)

There are no changes which would affect how a container runtime is resolved, which is what the error in this issue describes.

The features are 2 additional modules which do not affect the core. The bug fix resolves whether or not images should be force pulled, something which happens after a container runtime is resolved. The new SSHD image affects the port forwarding functionality which you do not use. The dependency updates are all minor. The core dependencies have not even been updated. These changes have all passed the builds and smoke tests.

I would like some more evidence that this is an issue with Testcontainers and not a change to your kubedock environment before investigating further.

[jonesbusy]

Sure I will provide evidence

Could it be minor dependencies upgraded in https://github.com/testcontainers/testcontainers-node/pull/780 ?

No change on the infrastructure. Basically renovate found the upgrade. This is the only package upgraded and it's dependencies

The main branch is building (I've run it again to be sure. All works with 10.9)

The only difference I can see on the DEBUG log is the ConfigurationStrategy that fail but doesn't give me the reason

On the renovate branch

On main branch with 10.9 everything is good

It happens on my 2 application using testcontainers (Spotify Backstage and one very simple Vite application). Same error

Despite

DEBUG = 'testcontainers*'

Do you think we can get more logs why the ConfigurationStrategy fail ?

Regards,

[rudemex]

I have the same problem

[zotijs]

@jonesbusy can you provide you gitlab ci job configuration? I have the same issue on 10.9.0 with @tesctontainers/mysql and with this job configuration:

test:
  stage: test
  extends: .test_application
  services:
    - name: docker:dind
      alias: docker
      # explicitly disable tls to avoid docker startup interruption
      command: ['--tls=false']
  variables:
    DOCKER_HOST: 'tcp://docker:2375'
    DOCKER_TLS_CERTDIR: ''
    DEBUG: testcontainers*
  only:
    - merge_requests

• Docker executor
• node lts alpine image

[cristianrgreco]

I've tried to reproduce this issue in GitLab and cannot:

---

.gitlab-ci.yml:

test:
  image: node:20-alpine
  services:
    - docker:dind
  variables:
    DOCKER_HOST: tcp://docker:2375/
    DOCKER_DRIVER: overlay2
    DEBUG: testcontainers*
  script:
    - npm install
    - npm test

test.ts:

import { GenericContainer } from "testcontainers";

describe("Test", () => {
  it("1", async () => {
    const container = await new GenericContainer("alpine").start();
    await container.stop();
  });
});

package.json dependencies:

"devDependencies": {
  "@types/jest": "^29.5.12",
  "jest": "^29.7.0",
  "testcontainers": "^10.10.0",
  "ts-jest": "^29.1.5",
  "typescript": "^5.5.2"
}

Could someone please help me get a reproducer?

[jonesbusy]

Hi @zotijs I'm not using gitlab. I have Jenkins and K8S agent and use https://github.com/joyrex2001/kubedock to proxy the docker socket call to Kubernetes and create testcontainers

The only vars I'm using on my Jenkinsfile is

DOCKER_HOST = 'tcp://localhost:2475'
TESTCONTAINERS_HUB_IMAGE_NAME_PREFIX  = '*********-docker-io.***************.ch/'
DEBUG = 'testcontainers*'

[zotijs]

@cristianrgreco I will try to create a reproduction repository, because the repository that I am having the issue is private

[jonesbusy]

Adding more logs I get

On 10.10.0

[11:18:31.368+02:00] - 2024-06-25T09:18:31.185Z testcontainers [DEBUG] Testing container runtime strategy "ConfigurationStrategy"...
[11:18:31.368+02:00] - 2024-06-25T09:18:31.197Z modem Sending: {
[11:18:31.368+02:00] -   path: '/info',
[11:18:31.368+02:00] -   method: 'GET',
[11:18:31.368+02:00] -   headers: { 'User-Agent': 'tc-node/10.10.0' },
[11:18:31.368+02:00] -   key: undefined,
[11:18:31.368+02:00] -   cert: undefined,
[11:18:31.368+02:00] -   ca: undefined,
[11:18:31.368+02:00] -   hostname: 'localhost',
[11:18:31.368+02:00] -   port: '2475'
[11:18:31.368+02:00] - }
[11:18:31.368+02:00] - 2024-06-25T09:18:31.205Z modem Received: {"ID":"com.joyrex2001.kubedock","Labels":[],"MemTotal":0,"Name":"kubedock","OperatingSystem":"kubernetes","ServerVersion":"0.16.0"}
[11:18:31.368+02:00] - 2024-06-25T09:18:31.230Z testcontainers [DEBUG] Container runtime strategy "ConfigurationStrategy" does not work

On 10.9.0

[12:59:11.118+02:00] - 2024-06-25T10:59:10.930Z modem Sending: {
[12:59:11.118+02:00] -   path: '/info',
[12:59:11.118+02:00] -   method: 'GET',
[12:59:11.118+02:00] -   headers: { 'User-Agent': 'tc-node/10.9.0' },
[12:59:11.118+02:00] -   key: undefined,
[12:59:11.118+02:00] -   cert: undefined,
[12:59:11.118+02:00] -   ca: undefined,
[12:59:11.118+02:00] -   hostname: 'localhost',
[12:59:11.118+02:00] -   port: '2475'
[12:59:11.118+02:00] - }
[12:59:11.118+02:00] - 2024-06-25T10:59:10.938Z modem Received: {"ID":"com.joyrex2001.kubedock","Labels":[],"MemTotal":0,"Name":"kubedock","OperatingSystem":"kubernetes","ServerVersion":"0.16.0"}
[12:59:11.118+02:00] - 2024-06-25T10:59:10.974Z testcontainers [DEBUG] Container runtime strategy "ConfigurationStrategy" works

Not sure it help.

Response is identical

[zotijs]

@cristianrgreco sorry for the delayed response. So a similar setup to what you have provided works for me too, gitlab-pipeline. I will create a branch with something similar to what I am actually doing in the private repo. But I am wondering if the issue is the group runners that we are using inside the organization. Anyhow I will come back with a more realistic example

[jonesbusy]

Hi,

This is the logs after testing the 10.10.1

[06:25:49.508+02:00] - 2024-07-08T04:25:49.272Z modem Received: {"ID":"com.joyrex2001.kubedock","Labels":[],"MemTotal":0,"Name":"kubedock","OperatingSystem":"kubernetes","ServerVersion":"0.16.0"}
[06:25:49.509+02:00] - 2024-07-08T04:25:49.274Z testcontainers [TRACE] Fetching remote container runtime socket path...
[06:25:49.509+02:00] - 2024-07-08T04:25:49.274Z testcontainers [TRACE] Resolving host...
[06:25:49.509+02:00] - 2024-07-08T04:25:49.274Z testcontainers [TRACE] Fetching Compose info...
[06:25:49.509+02:00] - 2024-07-08T04:25:49.298Z testcontainers [TRACE] Looking up host IPs...
[06:25:49.509+02:00] - 2024-07-08T04:25:49.300Z testcontainers [TRACE] Initialising clients...
[06:25:49.509+02:00] - 2024-07-08T04:25:49.301Z testcontainers [DEBUG] Container runtime strategy "ConfigurationStrategy" does not work: TypeError: Cannot convert undefined or null to object
[06:25:49.509+02:00] - 2024-07-08T04:25:49.301Z testcontainers [DEBUG] Checking container runtime strategy "UnixSocketStrategy"...
[06:25:49.509+02:00] - 2024-07-08T04:25:49.301Z testcontainers [DEBUG] Container runtime strategy "UnixSocketStrategy" is not applicable
[06:25:49.509+02:00] - 2024-07-08T04:25:49.301Z testcontainers [DEBUG] Checking container runtime strategy "RootlessUnixSocketStrategy"...
[06:25:49.509+02:00] - 2024-07-08T04:25:49.301Z testcontainers [DEBUG] Container runtime strategy "RootlessUnixSocketStrategy" is not applicable
[06:25:49.509+02:00] - 2024-07-08T04:25:49.301Z testcontainers [DEBUG] Checking container runtime strategy "NpipeSocketStrategy"...
[06:25:49.510+02:00] - 2024-07-08T04:25:49.302Z testcontainers [DEBUG] Container runtime strategy "NpipeSocketStrategy" is not applicable

Container runtime strategy "ConfigurationStrategy" does not work: TypeError: Cannot convert undefined or null to object

Any idea ?

[cristianrgreco]

@jonesbusy That's progress, seems there's some issue with the configuration used by the ConfigurationStrategy. Unfortunately the error doesn't pinpoint much so I've raised #794 to print a stacktrace if there is one. I'll release this as a patch soon and let you know and hopefully you won't mind retrying?

[jonesbusy]

@cristianrgreco Sure I will try as soon as it's released. Thanks

[cristianrgreco]

Thanks @jonesbusy. Released in 10.10.2

[jonesbusy]

This is the stackstrace of 10.10.2

[06:04:18.991+02:00] - 2024-07-09T04:04:18.874Z testcontainers [DEBUG] TypeError: Cannot convert undefined or null to object
[06:04:18.991+02:00] -     at Function.keys (<anonymous>)
[06:04:18.991+02:00] -     at initStrategy (/home/jenkins/agent/workspace/****-10.10.2/node_modules/testcontainers/src/container-runtime/clients/client.ts:125:22)
[06:04:18.991+02:00] -     at getContainerRuntimeClient (/home/jenkins/agent/workspace/*****-10.10.2/node_modules/testcontainers/src/container-runtime/clients/client.ts:51:22)
[06:04:18.991+02:00] -     at GenericContainer.start (/home/jenkins/agent/workspace/e_feature_test-containe*******r-10.10.2/node_modules/testcontainers/src/generic-container/generic-container.ts:81:20)
[06:04:18.991+02:00] -     at /home/jenkins/agent/workspace/****-10.10.2/src/ts/api.test.ts:23:25
[06:04:18.991+02:00] -     at async Promise.all (index 0)
[06:04:18.991+02:00] -     at callSuiteHook (file:///home/jenkins/agent/workspace/****-10.10.2/node_modules/@vitest/runner/dist/index.js:700:23)
[06:04:18.991+02:00] -     at runSuite (file:///home/jenkins/agent/workspace/****-10.10.2/node_modules/@vitest/runner/dist/index.js:892:27)
[06:04:18.991+02:00] -     at runSuite (file:///home/jenkins/agent/workspace/*****-10.10.2/node_modules/@vitest/runner/dist/index.js:909:15)
[06:04:18.991+02:00] -     at runFiles (file:///home/jenkins/agent/workspace/*******-10.10.2/node_modules/@vitest/runner/dist/index.js:958:5)
[06:04:18.991+02:00] - 2024-07-09T04:04:18.874Z testcontainers [DEBUG] Checking container runtime strategy "UnixSocketStrategy"...

[cristianrgreco]

Thanks @jonesbusy, turns out this is your issue: https://github.com/testcontainers/testcontainers-node/commit/ab4667329883d03f801fb7ecb7b3e86eabb1e10c#diff-07adb1d2976c5e124c3804996a054095cdc770c58ec8a1b7af5c24f37da21ee7R109, added as part of the Ollama module.

Will be fixed in #795.

Thanks for your help in debugging this one, I'm happy with the error logs we've achieved around the container runtime

[cristianrgreco]

Will leave issue open until released and fix confirmed. ETA for release within next few hours.

[cristianrgreco]

@jonesbusy Please try 10.10.3

[jonesbusy]

Just done

All work as expected

Thank you so much for the help!

[cristianrgreco]

Thanks to you for raising such a detailed issue and for helping so much in debugging, always a pleasure! 🙂