404 when starting a container

[dpp23]

trafficstars

Running a container build from the following Dockerfile:

            FROM alpine
            RUN echo \$RANDOM >> /tmp/test.txt
            CMD cat /tmp/test.txt && echo "DONE" && sleep 28800

I get 404 from when calling container.start. When I look at the pods inside k8s, everything looks good, so I think this is a bug inside kubedock.

In the logs I get the following, but not sure if it is a red herring:

[GIN-debug] [WARNING] Headers were already written. Wanted to override status code 404 with 204

I am going to try and figure out a small runnable example to demonstrate this.

[dpp23]

I can't get a reliable reproducible example so far, but when this fails the logs look like this:

I0302 12:45:11.323301     279 util.go:77] Request Headers: http.Header{"Accept":[]string{"application/json"}, "Accept-Encoding":[]string{"gzip, x-gzip, deflate"}, "Connection":[]string{"keep-alive"}, "Content-Length":[]string{"1968"}, "Content-Type":[]string{"application/json"}, "User-Agent":[]string{"Apache-HttpClient/5.0.3 (Java/11.0.8)"}}
I0302 12:45:11.323332     279 util.go:78] Request Body: {"name":null,"authConfig":null,"platform":null,"Hostname":null,"Domainname":null,"User":null,"AttachStdin":null,"AttachStdout":null,"AttachStderr":null,"PortSpecs":null,"Tty":null,"OpenStdin":null,"StdinOnce":null,"Env":[],"Cmd":[],"Healthcheck":null,"ArgsEscaped":null,"Entrypoint":null,"Image":"********************************************/buildsrc-functional-tests:snapshot-1be97a70dbf46c604c97-test-cache-2","Volumes":{},"WorkingDir":null,"MacAddress":null,"OnBuild":null,"NetworkDisabled":null,"ExposedPorts":{},"StopSignal":null,"StopTimeout":null,"HostConfig":{"Binds":[],"BlkioWeight":null,"BlkioWeightDevice":null,"BlkioDeviceReadBps":null,"BlkioDeviceWriteBps":null,"BlkioDeviceReadIOps":null,"BlkioDeviceWriteIOps":null,"MemorySwappiness":null,"NanoCpus":null,"CapAdd":null,"CapDrop":null,"ContainerIDFile":null,"CpuPeriod":null,"CpuRealtimePeriod":null,"CpuRealtimeRuntime":null,"CpuShares":null,"CpuQuota":null,"CpusetCpus":null,"CpusetMems":null,"Devices":null,"DeviceCgroupRules":null,"DeviceRequests":null,"DiskQuota":null,"Dns":null,"DnsOptions":null,"DnsSearch":null,"ExtraHosts":[],"GroupAdd":null,"IpcMode":null,"Cgroup":null,"Links":[],"LogConfig":null,"LxcConf":null,"Memory":null,"MemorySwap":null,"MemoryReservation":null,"KernelMemory":null,"NetworkMode":null,"OomKillDisable":null,"Init":null,"AutoRemove":null,"OomScoreAdj":null,"PortBindings":{},"Privileged":null,"PublishAllPorts":null,"ReadonlyRootfs":null,"RestartPolicy":null,"Ulimits":null,"CpuCount":null,"CpuPercent":null,"IOMaximumIOps":null,"IOMaximumBandwidth":null,"VolumesFrom":[],"Mounts":null,"PidMode":null,"Isolation":null,"SecurityOpt":null,"StorageOpt":null,"CgroupParent":null,"VolumeDriver":null,"ShmSize":null,"PidsLimit":null,"Runtime":null,"Tmpfs":null,"UTSMode":null,"UsernsMode":null,"Sysctls":null,"ConsoleSize":null},"Labels":{"org.testcontainers":"true","org.testcontainers.sessionId":"eb8b57c2-d565-436b-9b6d-d27b2bba68e9"},"Shell":null,"NetworkingConfig":null}
I0302 12:45:11.323457     279 util.go:101] Response Body: {"Id":"dfd5522dc93fd9df86dc09e478b1215b8d8cc5835f0a2495f02f24392cacf964"}
[GIN] 2022/03/02 - 12:45:11 | 201 |     190.096µs |       127.0.0.1 | POST     "/containers/create"
I0302 12:45:11.324698     279 util.go:77] Request Headers: http.Header{"Accept":[]string{"application/json"}, "Accept-Encoding":[]string{"gzip, x-gzip, deflate"}, "Connection":[]string{"keep-alive"}, "Content-Type":[]string{"application/json"}, "User-Agent":[]string{"Apache-HttpClient/5.0.3 (Java/11.0.8)"}}
I0302 12:45:11.324727     279 util.go:78] Request Body: 
W0302 12:45:11.324748     279 container.go:161] user not set, will run as user defined in image
I0302 12:45:11.373990     279 main.go:68] finished cleaning lingering objects...
[GIN-debug] [WARNING] Headers were already written. Wanted to override status code 404 with 204
I0302 12:45:14.819066     279 util.go:101] Response Body: 
[GIN] 2022/03/02 - 12:45:14 | 204 |  3.494384342s |       127.0.0.1 | POST     "/containers/dfd5522dc93fd9df86dc09e478b1215b8d8cc5835f0a2495f02f24392cacf964/start"

But testcontainers fails with:

Could not start container

    com.github.dockerjava.api.exception.NotFoundException: Status 404

which makes me think that despite what the log says, POST "/containers/dfd5522dc93fd9df86dc09e478b1215b8d8cc5835f0a2495f02f24392cacf964/start returned 404

Something else to note is that the image is build and pushed just before running the test.

[dpp23]

At the point we get to this line

c.Writer.Written() is true and c.Writer.Status() is 404. Not sure what sets them though.

[joyrex2001]

Not sure either, maybe it's the verbosity flag, does it occur with -v 1 as well?

[joyrex2001]

Is this still an issue? The orchestration of containers has been changed in 0.11.0, including the handling of completed containers and checking if a container was started succesfully.

[dpp23]

I don't know. Our use case for using kubedock was removed some time ago.

Given that there is little to go on from my original bug report, probably not worth keeping this open