The unhandled exception could crash the deck-chores container

[vitoyucepi]

Summary

I found a case when managing the containers labels related to deck-chores could trigger various exceptions.

This exception causes the deck-chores container to crash.

Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/docker/api/client.py", line 268, in _raise_for_status
    response.raise_for_status()
  File "/usr/local/lib/python3.11/site-packages/requests/models.py", line 1021, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 404 Client Error: Not Found for url: http+docker://localhost/v1.41/containers/af2812dcb93a9b79e1460e7ce058887fe1e785e05868b46666780830d575fca2/json

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/deck_chores/main.py", line 320, in main
    listen(since=last_event_time + timedelta(microseconds=1))
  File "/usr/local/lib/python3.11/site-packages/deck_chores/main.py", line 204, in listen
    handle_die(event)
  File "/usr/local/lib/python3.11/site-packages/deck_chores/main.py", line 220, in handle_die
    if reassign_jobs(container_id, consider_paused=True) is None:
       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/deck_chores/main.py", line 132, in reassign_jobs
    log.info(f"{container_name(container_id)}: Reassigning jobs to {new_id}.")
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/deck_chores/indexes.py", line 13, in container_name
    return cfg.client.containers.get(container_id).name
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/docker/models/containers.py", line 925, in get
    resp = self.client.api.inspect_container(container_id)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/docker/utils/decorators.py", line 19, in wrapped
    return f(self, resource_id, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/docker/api/container.py", line 783, in inspect_container
    return self._result(
           ^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/docker/api/client.py", line 274, in _result
    self._raise_for_status(response)
  File "/usr/local/lib/python3.11/site-packages/docker/api/client.py", line 270, in _raise_for_status
    raise create_api_error_from_http_exception(e) from e
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/docker/errors.py", line 39, in create_api_error_from_http_exception
    raise cls(e, response=response, explanation=explanation) from e
docker.errors.NotFound: 404 Client Error for http+docker://localhost/v1.41/containers/af2812dcb93a9b79e1460e7ce058887fe1e785e05868b46666780830d575fca2/json: Not Found ("No such container: af2812dcb93a9b79e1460e7ce058887fe1e785e05868b46666780830d575fca2")

Reproduction

1. Start deck-chores docker run -it -v /var/run/docker.sock:/var/run/docker.sock -e TZ=UTC funkyfuture/deck-chores:1.3.10.
2. Create two compose configs for the same server. compose1.yaml

   services:
     test:
       image: alpine:3.17
       labels:
         deck-chores.test.command: env
         deck-chores.test.interval: every minute
       command: sleep 1000
       stop_grace_period: 1s
   

   compose2.yaml

   services:
     test:
       image: alpine:3.17
       labels:
         deck-chores.test.command: env
         deck-chores.test.interval: every second
       command: sleep 1000
       stop_grace_period: 1s
   

3. Run command docker-compose -f compose1.yaml up -d; docker-compose -f compose2.yaml up -d; docker-compose -f compose1.yaml up -d.

Version

• docker: 20.10.23
• docker-compose: 2.16.0
• deck-chores: 1.3.10

Additional

There are also other exceptions, but they don't terminate the container.

exception 1

Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/docker/api/client.py", line 268, in _raise_for_status
    response.raise_for_status()
  File "/usr/local/lib/python3.11/site-packages/requests/models.py", line 1021, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 404 Client Error: Not Found for url: http+docker://localhost/v1.41/containers/5b4187e9c9f99aba5ea729006bdb602f23cad836340af557841907c99669a315/exec

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/apscheduler/executors/base.py", line 125, in run_job
    retval = job.func(*job.args, **job.kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/deck_chores/jobs.py", line 103, in exec_job
    return cfg.client.containers.get(container_id).exec_run(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/docker/models/containers.py", line 193, in exec_run
    resp = self.client.api.exec_create(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/docker/utils/decorators.py", line 19, in wrapped
    return f(self, resource_id, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/docker/api/exec_api.py", line 78, in exec_create
    return self._result(res, True)
           ^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/docker/api/client.py", line 274, in _result
    self._raise_for_status(response)
  File "/usr/local/lib/python3.11/site-packages/docker/api/client.py", line 270, in _raise_for_status
    raise create_api_error_from_http_exception(e) from e
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/docker/errors.py", line 39, in create_api_error_from_http_exception
    raise cls(e, response=response, explanation=explanation) from e
docker.errors.NotFound: 404 Client Error for http+docker://localhost/v1.41/containers/5b4187e9c9f99aba5ea729006bdb602f23cad836340af557841907c99669a315/exec: Not Found ("No such container: 5b4187e9c9f99aba5ea729006bdb602f23cad836340af557841907c99669a315")

Error notifying listener
Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/apscheduler/schedulers/base.py", line 835, in _dispatch_event
    cb(event)
  File "/usr/local/lib/python3.11/site-packages/deck_chores/jobs.py", line 67, in on_error
    definition = scheduler.get_job(event.job_id).kwargs
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AttributeError: 'NoneType' object has no attribute 'kwargs'

exception 2

Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/docker/api/client.py", line 268, in _raise_for_status
    response.raise_for_status()
  File "/usr/local/lib/python3.11/site-packages/requests/models.py", line 1021, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 404 Client Error: Not Found for url: http+docker://localhost/v1.41/containers/771566280f258ced0e81f1e5128ef7e3872e633ef316cc3785374e3219eda983/exec

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/apscheduler/executors/base.py", line 125, in run_job
    retval = job.func(*job.args, **job.kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/deck_chores/jobs.py", line 103, in exec_job
    return cfg.client.containers.get(container_id).exec_run(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/docker/models/containers.py", line 193, in exec_run
    resp = self.client.api.exec_create(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/docker/utils/decorators.py", line 19, in wrapped
    return f(self, resource_id, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/docker/api/exec_api.py", line 78, in exec_create
    return self._result(res, True)
           ^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/docker/api/client.py", line 274, in _result
    self._raise_for_status(response)
  File "/usr/local/lib/python3.11/site-packages/docker/api/client.py", line 270, in _raise_for_status
    raise create_api_error_from_http_exception(e) from e
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/docker/errors.py", line 39, in create_api_error_from_http_exception
    raise cls(e, response=response, explanation=explanation) from e
docker.errors.NotFound: 404 Client Error for http+docker://localhost/v1.41/containers/771566280f258ced0e81f1e5128ef7e3872e633ef316cc3785374e3219eda983/exec: Not Found ("No such container: 771566280f258ced0e81f1e5128ef7e3872e633ef316cc3785374e3219eda983")

exception 3

Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/docker/api/client.py", line 268, in _raise_for_status
    response.raise_for_status()
  File "/usr/local/lib/python3.11/site-packages/requests/models.py", line 1021, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 409 Client Error: Conflict for url: http+docker://localhost/v1.41/containers/49578007e090ab7dede4e73bec55a019afc6ed5a7e0dad045e7054c293923e3f/exec

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/apscheduler/executors/base.py", line 125, in run_job
    retval = job.func(*job.args, **job.kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/deck_chores/jobs.py", line 103, in exec_job
    return cfg.client.containers.get(container_id).exec_run(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/docker/models/containers.py", line 193, in exec_run
    resp = self.client.api.exec_create(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/docker/utils/decorators.py", line 19, in wrapped
    return f(self, resource_id, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/docker/api/exec_api.py", line 78, in exec_create
    return self._result(res, True)
           ^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/docker/api/client.py", line 274, in _result
    self._raise_for_status(response)
  File "/usr/local/lib/python3.11/site-packages/docker/api/client.py", line 270, in _raise_for_status
    raise create_api_error_from_http_exception(e) from e
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/docker/errors.py", line 39, in create_api_error_from_http_exception
    raise cls(e, response=response, explanation=explanation) from e
docker.errors.APIError: 409 Client Error for http+docker://localhost/v1.41/containers/49578007e090ab7dede4e73bec55a019afc6ed5a7e0dad045e7054c293923e3f/exec: Conflict ("Container 49578007e090ab7dede4e73bec55a019afc6ed5a7e0dad045e7054c293923e3f is restarting, wait until the container is running")