[Bug] - Containers such as Fedora 40 not starting after upgrade to workspaces 1.16

[luc-vocab]

Existing Resources

• [x] Please search the existing issues for related problems
• [x] Consult the product documentation : Docs
• [x] Consult the FAQ : FAQ
• [x] Consult the Troubleshooting Guide : Guide
• [x] Reviewed existing training videos: Youtube

Describe the bug After upgrading to Kasm Workspaces 1.16, some containers such as Fedora 40 are not starting up. Here are logs from the docker daemon:

ep 30 19:21:58 workspaces dockerd[43508]: time="2024-09-30T19:21:58.228816615+08:00" level=info msg="ignoring event" container=883200152b997a3fe0c8a591acdc42c095991aef37361648917e284dae404261 module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
Sep 30 19:21:58 workspaces dockerd[43508]: time="2024-09-30T19:21:58.326355900+08:00" level=error msg="Error setting up exec command in container 883200152b997a3fe0c8a591acdc42c095991aef37361648917e284dae404261: Container 883200152b997a3fe0c8a591acdc42c095991aef37361648917e284dae404261 is restarting, wait until the container is running" spanID=c8e720c5e0edaecc traceID=a2a91cc25b7c31e079afa7b8668cacf6
Sep 30 19:21:58 workspaces dockerd[43508]: time="2024-09-30T19:21:58.328944607+08:00" level=error msg="Error setting up exec command in container 883200152b997a3fe0c8a591acdc42c095991aef37361648917e284dae404261: Container 883200152b997a3fe0c8a591acdc42c095991aef37361648917e284dae404261 is restarting, wait until the container is running" spanID=06f31766f72d9402 traceID=465405bce2618a4142f0eefd1ff51666
Sep 30 19:21:58 workspaces dockerd[43508]: time="2024-09-30T19:21:58.478565080+08:00" level=info msg="ignoring event" container=883200152b997a3fe0c8a591acdc42c095991aef37361648917e284dae404261 module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
Sep 30 19:21:58 workspaces dockerd[43508]: time="2024-09-30T19:21:58.486317974+08:00" level=warning msg="ShouldRestart failed, container will not be restarted" container=883200152b997a3fe0c8a591acdc42c095991aef37361648917e284dae404261 daemonShuttingDown=false error="restart canceled" execDuration=123.597838ms exitStatus="{137 2024-09-30 11:21:58.472815308 +0000 UTC}" hasBeenManuallyStopped=true restartCount=1

Seeing the following error in the Kasm Workspaces logging page in the web interface:

host: workspaces
ingest_date: 202409301121
application: kasm_api
levelname: ERROR
kasm_user_name: luc
process: client_api_server
client_ip: [2001:470:f82b:a::6]:53154
user_agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/128.0.0.0 Safari/537.36
message
An Unexpected Error occurred creating the Kasm. Please contact an Administrator : Error during Create request for Server(14cc3c29-e0ac-451a-9523-d632832efdd6) : (Exception creating Kasm: Traceback (most recent call last):
  File "docker/api/client.py", line 265, in _raise_for_status
  File "requests/models.py", line 1021, in raise_for_status
requests.exceptions.HTTPError: 409 Client Error: Conflict for url: http+docker://localhost/v1.47/containers/883200152b997a3fe0c8a591acdc42c095991aef37361648917e284dae404261/exec

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "__init__.py", line 574, in post
  File "provision.py", line 1943, in provision
  File "docker/models/containers.py", line 205, in exec_run
  File "docker/utils/decorators.py", line 19, in wrapped
  File "docker/api/exec_api.py", line 79, in exec_create
  File "docker/api/client.py", line 271, in _result
  File "docker/api/client.py", line 267, in _raise_for_status
  File "docker/errors.py", line 39, in create_api_error_from_http_exception
docker.errors.APIError: 409 Client Error for http+docker://localhost/v1.47/containers/883200152b997a3fe0c8a591acdc42c095991aef37361648917e284dae404261/exec: Conflict ("Container 883200152b997a3fe0c8a591acdc42c095991aef37361648917e284dae404261 is restarting, wait until the container is running")
)

To Reproduce Start Fedora 40 container.

Expected behavior I expect the container to start up.

Workspaces Version 1.126

Workspaces Installation Method Single Server

Client Browser (please complete the following information):

• OS: Fedora Linux 40
• Browser: Chromium
• Version 128

Workspace Server Information (please provide the output of the following commands):

• uname -a:
  Linux workspaces 6.8.0-45-generic #45-Ubuntu SMP PREEMPT_DYNAMIC Fri Aug 30 12:02:04 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
• cat /etc/os-release

 PRETTY_NAME="Ubuntu 24.04.1 LTS"
NAME="Ubuntu"
VERSION_ID="24.04"
VERSION="24.04.1 LTS (Noble Numbat)"
VERSION_CODENAME=noble
ID=ubuntu
ID_LIKE=debian
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
UBUNTU_CODENAME=noble
LOGO=ubuntu-logo

• sudo docker info

lient: Docker Engine - Community
 Version:    27.3.1
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.17.1
    Path:     /usr/libexec/docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v2.29.7
    Path:     /usr/libexec/docker/cli-plugins/docker-compose

Server:
 Containers: 10
  Running: 10
  Paused: 0
  Stopped: 0
 Images: 30
 Server Version: 27.3.1
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: systemd
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan kasmweb/sidecar:1.0 macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 7f7fdf5fed64eb6a7caf99b3e12efcf9d60e311c
 runc version: v1.1.14-0-g2c9f560
 init version: de40ad0
 Security Options:
  apparmor
  seccomp
   Profile: builtin
  cgroupns
 Kernel Version: 6.8.0-45-generic
 Operating System: Ubuntu 24.04.1 LTS
 OSType: linux
 Architecture: x86_64
 CPUs: 16
 Total Memory: 7.614GiB
 Name: workspaces
 ID: 84f43ca8-8c93-4f4f-84f8-dcb5f136f936
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Experimental: true
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

• sudo docker ps | grep kasm

89dad3882bfc   kasmweb/proxy:1.16.0               "/docker-entrypoint.…"   8 hours ago   Up 7 hours             80/tcp, 0.0.0.0:8443->8443/tcp, :::8443->8443/tcp   kasm_proxy
a2529adeb1b2   kasmweb/rdp-https-gateway:1.16.0   "/opt/rdpgw/rdpgw"       8 hours ago   Up 7 hours (healthy)                                                       kasm_rdp_https_gateway
d8143bace463   kasmweb/rdp-gateway:1.16.0         "/start.sh"              8 hours ago   Up 7 hours (healthy)   0.0.0.0:3389->3389/tcp, :::3389->3389/tcp           kasm_rdp_gateway
aedce1a2de25   kasmweb/agent:1.16.0               "/bin/sh -c '/usr/bi…"   8 hours ago   Up 7 hours (healthy)   4444/tcp                                            kasm_agent
5e7c81e9a697   kasmweb/share:1.16.0               "/bin/sh -c '/usr/bi…"   8 hours ago   Up 7 hours (healthy)   8182/tcp                                            kasm_share
8bb11f7b8748   kasmweb/manager:1.16.0             "/usr/bin/startup.sh…"   8 hours ago   Up 7 hours (healthy)   8181/tcp                                            kasm_manager
7a3906b7df5d   kasmweb/api:1.16.0                 "/bin/sh -c '/usr/bi…"   8 hours ago   Up 7 hours (healthy)   8080/tcp                                            kasm_api
94c7ea14ba1e   kasmweb/kasm-guac:1.16.0           "/dockerentrypoint.sh"   8 hours ago   Up 7 hours (healthy)                                                       kasm_guac
1a7d6dc34a1a   redis:5-alpine                     "docker-entrypoint.s…"   8 hours ago   Up 7 hours             6379/tcp                                            kasm_redis
803bf0279fc1   postgres:14-alpine                 "docker-entrypoint.s…"   8 hours ago   Up 7 hours (healthy)   5432/tcp                                            kasm_db

Additional context Add any other context about the problem here.