finch
finch copied to clipboard
runfinch
Publishing of the ports doesn't work after a the container restart.
Bug Description Finch doesn't publish the ports after restarting the container.
Steps to reproduce
- In Terminal run
finch run -it -p 3003:80 nginxdemos/hello - In another terminal tab run
curl http://localhost:3003/. This should return some HTML - Stop the original docker container by pressing
CTRL+Cin the finch terminal tab - Start the docker container again by running
finch run -it -p 3003:80 nginxdemos/hello. This will print some kind of error in the terminal similar to:
FATA[0000] failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: error running hook #0: error running hook: exit status 1, stdout: , stderr: time="2023-01-17T10:56:31Z" level=fatal msg="conflict with ID 8"
Failed to write to log, write /home/tmikus.linux/.local/share/nerdctl/1935db59/containers/finch/29d8289e7ad41be113b94c8738ac2d690ea5aea16f0f9fd047c98e5aa8ee51e0/oci-hook.createRuntime.log: file already closed: unknown
FATA[0000] exit status 1
- Start the container again by running
finch run -it -p 3003:80 nginxdemos/hello. This should correctly start nginx - In another tab run
curl http://localhost:3003/. This will hang and might returncurl: (52) Empty reply from serverafter 30 seconds
Expected behaviour
- Finch shouldn't return an error on step 4
- Finch should publish the port 3003 on step 6 and should allow for the
curlto return a result on step 7
Workaround Currently, the only workaround I know for this is to restart the finch vm before starting the docker container:
finch vm stop && finch vm start
Thanks for reporting this bug. finch wraps on nerdctl which does most of the "heavy lifting". I noticed this is happening in nerdctl. I have opened an issue in their queue. We will continue with our investigation and post any updates here.
I think there are multiple nuances to this issue, the originally reported issue is not an issue about container restarts, but instead showing an error when creating a new container on the same port of a stopped container. This highlights a couple of bugs, and a different UX to the Docker Engine, so I have gone through multiple scenarios and here are my results:
Tests were carried out on Finch v0.4.0 on MacOS 12.6.3
- Starting multiple running containers on the same port should consistently error out. This works as expected for
test2however it lets me starttest3. That is a bug, Finch should have preventedtest3from being started.
$ finch run --detach --name test1 --publish 80:80 nginx
9fda2e5bb35edb963c08ab8b81a0af02528725e86b812f2c96b32a5d524fcf41
$ finch run --detach --name test2 --publish 80:80 nginx
FATA[0000] failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: error running hook #0: error running hook: exit status 1, stdout: , stderr: time="2023-02-22T10:27:25Z" level=fatal msg="conflict with ID 4"
Failed to write to log, write /home/myuser.linux/.local/share/nerdctl/1935db59/containers/finch/0e05c244b7aa8db0e0cdf01bb3973698d40464102f43410363922a7a78564162/oci-hook.createRuntime.log: file already closed: unknown
FATA[0000] exit status 1
$ finch run --detach --name test3 --publish 80:80 nginx
867fa811b5f52de805d6d46953a41319bb7c721634f7c543799486caa17facd4
$ finch ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
0e05c244b7aa docker.io/library/nginx:latest "/docker-entrypoint.…" 24 seconds ago Created 0.0.0.0:80->80/tcp test2
867fa811b5f5 docker.io/library/nginx:latest "/docker-entrypoint.…" 19 seconds ago Up 0.0.0.0:80->80/tcp test3
9fda2e5bb35e docker.io/library/nginx:latest "/docker-entrypoint.…" 37 seconds ago Up 0.0.0.0:80->80/tcp test1
-
Attempting to use the same port as a stopped container (recreating the originally reported issue). Here there are 2 issues:
-
Finch prevents you from starting a new container on the same port as a stopped container. This is a different UX from the Docker Engine were you are able to start a new container on the same port as a stopped container. If we had consistent behavior with the Docker Engine,
test2should have started. -
Finch allows you to start a new container on the port of the stopped container via brute force (2 attempts).
test3's port is unreachable even though it is started successfully, so I don't believe this 3rd container is properly wired up. This is a bug as Finch should have either preventedtest3from being started, or if it letstest3start, wire up the networking correctly.
-
$ finch run --detach --name test1 --publish 80:80 nginx
b0f1a860e060b859b1ded9206e218f3bd3200ba707111be9fe984243e19b55f4
$ finch stop test1
test1
$ finch run --detach --name test2 --publish 80:80 nginx
FATA[0000] failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: error running hook #0: error running hook: exit status 1, stdout: , stderr: time="2023-02-22T10:25:24Z" level=fatal msg="conflict with ID 2"
Failed to write to log, write /home/myuser.linux/.local/share/nerdctl/1935db59/containers/finch/50bfe5531cb592704abf835778ebe9d39b25f77e4250fe010c9f00309805ef7f/oci-hook.createRuntime.log: file already closed: unknown
FATA[0000] exit status 1
$ finch run --detach --name test3 --publish 80:80 nginx
39303ebec89e5dac7d6f42c0acfd0f510022d47bbe9884474681d71b55cdfd03
$ finch ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
39303ebec89e docker.io/library/nginx:latest "/docker-entrypoint.…" 8 seconds ago Up 0.0.0.0:80->80/tcp test3
50bfe5531cb5 docker.io/library/nginx:latest "/docker-entrypoint.…" 56 seconds ago Created 0.0.0.0:80->80/tcp test2
b0f1a860e060 docker.io/library/nginx:latest "/docker-entrypoint.…" About a minute ago Exited (0) About a minute ago 0.0.0.0:80->80/tcp test1
$
- Just testing a container restart (the title of the issue). This seems to work ok without an issue.
$ finch run --detach --name test1 --publish 80:80 nginx
29305494d261e5206286154d6becb9799e1582fdb8192fb2f0c99bcfb40d203b
$ finch stop test1
test1
$ finch start test1
test1
$ finch ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
29305494d261 docker.io/library/nginx:latest "/docker-entrypoint.…" 15 seconds ago Up 0.0.0.0:80->80/tcp test1
- However I think there is a related bug here on container restarts. It appears Finch is not freeing up the port when you are stopping / starting the VM. This can be shown when trying to stop / start an existing container after restarting the VM. Through brute force (3 attempts) I can get the container to restart correctly.
$ finch run --detach --name test1 --publish 80:80 nginx
29305494d261e5206286154d6becb9799e1582fdb8192fb2f0c99bcfb40d203b
$ finch ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
29305494d261 docker.io/library/nginx:latest "/docker-entrypoint.…" About a minute ago Up 0.0.0.0:80->80/tcp test1
# Restart the VM
$ finch vm stop
INFO[0000] Stopping existing Finch virtual machine...
INFO[0002] Finch virtual machine stopped successfully
$ finch vm start
INFO[0000] Starting existing Finch virtual machine...
INFO[0022] Finch virtual machine started successfully
# Container is stopped when the VM stops.
$ finch ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
29305494d261 docker.io/library/nginx:latest "/docker-entrypoint.…" About a minute ago Created 0.0.0.0:80->80/tcp test1
# Attempt 1
$ finch start test1
FATA[0000] failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: error running hook #0: error running hook: exit status 1, stdout: , stderr: time="2023-02-22T10:32:37Z" level=fatal msg="failed to call cni.Setup: plugin type=\"bridge\" failed (add): failed to allocate for range 0: 10.4.0.28 has been allocated to finch-29305494d261e5206286154d6becb9799e1582fdb8192fb2f0c99bcfb40d203b, duplicate allocation is not allowed"
Failed to write to log, write /home/myuser.linux/.local/share/nerdctl/1935db59/containers/finch/29305494d261e5206286154d6becb9799e1582fdb8192fb2f0c99bcfb40d203b/oci-hook.createRuntime.log: file already closed: unknown
FATA[0000] exit status 1
# Attempt 2
$ finch start test1
FATA[0000] failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: error running hook #0: error running hook: exit status 1, stdout: , stderr: time="2023-02-22T10:32:47Z" level=fatal msg="failed to call cni.Setup: plugin type=\"bridge\" failed (add): failed to set bridge addr: could not set bridge's mac: invalid argument"
Failed to write to log, write /home/myuser.linux/.local/share/nerdctl/1935db59/containers/finch/29305494d261e5206286154d6becb9799e1582fdb8192fb2f0c99bcfb40d203b/oci-hook.createRuntime.log: file already closed: unknown
FATA[0000] exit status 1
# Attempt 3. Success.
$ finch start test1
test1
$ finch ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
29305494d261 docker.io/library/nginx:latest "/docker-entrypoint.…" 2 minutes ago Up 0.0.0.0:80->80/tcp test1
Test run on finch version 0.5.0 on macOS 13.2.1
TL;DR: Switching to rootful containerd causes a change in behavior for issues 1 and 2 described in the linked GitHub issue, but these issues are still not resolved. However, issue 4 has been fixed in rootful mode.
Starting multiple running containers on the same port should consistently error out. This works as expected for test2 however it lets me start test3. That is a bug, Finch should have prevented test3 from being started. This is still a bug and it should not allow multiple containers to use the same port.
In rootful containerd even test2 does not complain and is in running state. This is a bug as we don't want multiple containers to try to bind to the same port.
% finch run --detach --name test1 --publish 80:80 nginx
1e09a207bba6b0129c69f88a2a2f2bfe6f4b369a763c12f8272951a34639f469
% finch run --detach --name test2 --publish 80:80 nginx
190d2d5118d2700fd22740875518cc1c22e0203f139fc1e4ad4c079ed0a0facd
% finch ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
190d2d5118d2 docker.io/library/nginx:latest "/docker-entrypoint.…" About a minute ago Up 0.0.0.0:80->80/tcp test2
1e09a207bba6 docker.io/library/nginx:latest "/docker-entrypoint.…" 2 minutes ago Up 0.0.0.0:80->80/tcp test1
Attempting to use the same port as a stopped container (recreating the originally reported issue). Here there are 2 issues:
After switching to rootful containerd finch allows you to start a new container on the port of the stopped container without brute force. However the container is not wired correctly and can't be reached from the host.
% finch run --detach --name test1 --publish 80:80 nginx
689ba20973837da8b825900c38b192264af71082847e831ec943d1d2417ded46
% finch stop test1
test1
% finch run --detach --name test2 --publish 80:80 nginx
f47bded4b92e5f63ec98c47566e450c4d50d1b72ceeb26d5319df0eed248e870
% curl localhost:80
curl: (7) Failed to connect to localhost port 80 after 17 ms: Connection refused
Just testing a container restart (the title of the issue). This seems to work ok without an issue.
This works fine without any error in rootful mode as well.
However I think there is a related bug here on container restarts. It appears Finch is not freeing up the port when you are stopping / starting the VM. This can be shown when trying to stop / start an existing container after restarting the VM. Through brute force (3 attempts) I can get the container to restart correctly.
This is fixed by rootful.
% finch vm stop
INFO[0000] Stopping existing Finch virtual machine...
INFO[0006] Finch virtual machine stopped successfully
% finch vm start
INFO[0000] Starting existing Finch virtual machine...
INFO[0050] Finch virtual machine started successfully
% finch ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
689ba2097383 docker.io/library/nginx:latest "/docker-entrypoint.…" 11 minutes ago Created 0.0.0.0:80->80/tcp test1
% finch start test1
test1