salt icon indicating copy to clipboard operation
salt copied to clipboard

[BUG] docker broken in 3007.2 with multiple exceptions

Open darkpixel opened this issue 7 months ago • 5 comments

Description Running any state that involves docker_image or docker_container crashes spectacularly.

Setup Create a state:

# dockertest.sls
helloworld_image:
  docker_image.present:
    - name: hello-world
    - tag: latest
    - force: true

helloworld:
  docker_container.running:
    - name: helloworld
    - image: hello-world:latest
    - auto_remove: True

Please be as specific as possible and give set-up details.

  • [X] on-prem machine
  • [ ] VM (Virtualbox, KVM, etc. please specify)
  • [ ] VM running on a cloud service, please be explicit and add details
  • [ ] container (Kubernetes, Docker, containerd, etc. please specify)
  • [ ] or a combination, please be explicit
  • [ ] jails if it is FreeBSD
  • [ ] classic packaging
  • [ ] onedir packaging
  • [ ] used bootstrap to install

Running:

No LSB modules are available.
Distributor ID:	Debian
Description:	Debian GNU/Linux 12 (bookworm)
Release:	12
Codename:	bookworm

Installed the Debian salt-minion package from Broadcom.

Steps to Reproduce the behavior Apply the state:

# salt-call state.sls dockertest

local:
----------
          ID: helloworld_image
    Function: docker_image.present
        Name: hello-world
      Result: False
     Comment: An exception occurred in this state: Traceback (most recent call last):
                File "/opt/saltstack/salt/lib/python3.10/site-packages/requests/adapters.py", line 633, in send
                  conn = self.get_connection_with_tls_context(
                File "/opt/saltstack/salt/lib/python3.10/site-packages/requests/adapters.py", line 489, in get_connection_with_tls_context
                  conn = self.poolmanager.connection_from_host(
                File "/opt/saltstack/salt/lib/python3.10/site-packages/urllib3/poolmanager.py", line 246, in connection_from_host
                  return self.connection_from_context(request_context)
                File "/opt/saltstack/salt/lib/python3.10/site-packages/urllib3/poolmanager.py", line 258, in connection_from_context
                  raise URLSchemeUnknown(scheme)
              urllib3.exceptions.URLSchemeUnknown: Not supported URL scheme http+docker
              
              During handling of the above exception, another exception occurred:
              
              Traceback (most recent call last):
                File "/opt/saltstack/salt/extras-3.10/docker/api/client.py", line 213, in _retrieve_server_version
                  return self.version(api_version=False)["ApiVersion"]
                File "/opt/saltstack/salt/extras-3.10/docker/api/daemon.py", line 181, in version
                  return self._result(self._get(url), json=True)
                File "/opt/saltstack/salt/extras-3.10/docker/utils/decorators.py", line 44, in inner
                  return f(self, *args, **kwargs)
                File "/opt/saltstack/salt/extras-3.10/docker/api/client.py", line 236, in _get
                  return self.get(url, **self._set_request_timeout(kwargs))
                File "/opt/saltstack/salt/lib/python3.10/site-packages/requests/sessions.py", line 602, in get
                  return self.request("GET", url, **kwargs)
                File "/opt/saltstack/salt/lib/python3.10/site-packages/requests/sessions.py", line 589, in request
                  resp = self.send(prep, **send_kwargs)
                File "/opt/saltstack/salt/lib/python3.10/site-packages/requests/sessions.py", line 703, in send
                  r = adapter.send(request, **kwargs)
                File "/opt/saltstack/salt/lib/python3.10/site-packages/requests/adapters.py", line 637, in send
                  raise InvalidURL(e, request=request)
              requests.exceptions.InvalidURL: Not supported URL scheme http+docker
              
              The above exception was the direct cause of the following exception:
              
              Traceback (most recent call last):
                File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/state.py", line 2469, in call
                  ret = self.states[cdata["full"]](
                File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/loader/lazy.py", line 160, in __call__
                  ret = self.loader.run(run_func, *args, **kwargs)
                File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/loader/lazy.py", line 1269, in run
                  return self._last_context.run(self._run_as, _func_or_method, *args, **kwargs)
                File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/loader/lazy.py", line 1284, in _run_as
                  ret = _func_or_method(*args, **kwargs)
                File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/loader/lazy.py", line 1320, in wrapper
                  return f(*args, **kwargs)
                File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/utils/decorators/extension_deprecation.py", line 39, in wrapper
                  return function(*args, **salt.utils.args.clean_kwargs(**kwargs))
                File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/states/docker_image.py", line 254, in present
                  image_info = __salt__["docker.inspect_image"](full_image)
                File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/loader/lazy.py", line 160, in __call__
                  ret = self.loader.run(run_func, *args, **kwargs)
                File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/loader/lazy.py", line 1269, in run
                  return self._last_context.run(self._run_as, _func_or_method, *args, **kwargs)
                File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/loader/lazy.py", line 1284, in _run_as
                  ret = _func_or_method(*args, **kwargs)
                File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/utils/decorators/extension_deprecation.py", line 39, in wrapper
                  return function(*args, **salt.utils.args.clean_kwargs(**kwargs))
                File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/modules/dockermod.py", line 1994, in inspect_image
                  ret = _client_wrapper("inspect_image", name)
                File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/modules/dockermod.py", line 456, in wrapper
                  __context__["docker.client"] = _get_client(timeout=timeout, **kwargs)
                File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/modules/dockermod.py", line 416, in _get_client
                  ret = docker.APIClient(**client_kwargs)
                File "/opt/saltstack/salt/extras-3.10/docker/api/client.py", line 197, in __init__
                  self._version = self._retrieve_server_version()
                File "/opt/saltstack/salt/extras-3.10/docker/api/client.py", line 220, in _retrieve_server_version
                  raise DockerException(
              docker.errors.DockerException: Error while fetching server API version: Not supported URL scheme http+docker
     Started: 11:05:22.158301
    Duration: 4.835 ms
     Changes:   
----------
          ID: helloworld
    Function: docker_container.running
      Result: False
     Comment: An exception occurred in this state: Traceback (most recent call last):
                File "/opt/saltstack/salt/lib/python3.10/site-packages/requests/adapters.py", line 633, in send
                  conn = self.get_connection_with_tls_context(
                File "/opt/saltstack/salt/lib/python3.10/site-packages/requests/adapters.py", line 489, in get_connection_with_tls_context
                  conn = self.poolmanager.connection_from_host(
                File "/opt/saltstack/salt/lib/python3.10/site-packages/urllib3/poolmanager.py", line 246, in connection_from_host
                  return self.connection_from_context(request_context)
                File "/opt/saltstack/salt/lib/python3.10/site-packages/urllib3/poolmanager.py", line 258, in connection_from_context
                  raise URLSchemeUnknown(scheme)
              urllib3.exceptions.URLSchemeUnknown: Not supported URL scheme http+docker
              
              During handling of the above exception, another exception occurred:
              
              Traceback (most recent call last):
                File "/opt/saltstack/salt/extras-3.10/docker/api/client.py", line 213, in _retrieve_server_version
                  return self.version(api_version=False)["ApiVersion"]
                File "/opt/saltstack/salt/extras-3.10/docker/api/daemon.py", line 181, in version
                  return self._result(self._get(url), json=True)
                File "/opt/saltstack/salt/extras-3.10/docker/utils/decorators.py", line 44, in inner
                  return f(self, *args, **kwargs)
                File "/opt/saltstack/salt/extras-3.10/docker/api/client.py", line 236, in _get
                  return self.get(url, **self._set_request_timeout(kwargs))
                File "/opt/saltstack/salt/lib/python3.10/site-packages/requests/sessions.py", line 602, in get
                  return self.request("GET", url, **kwargs)
                File "/opt/saltstack/salt/lib/python3.10/site-packages/requests/sessions.py", line 589, in request
                  resp = self.send(prep, **send_kwargs)
                File "/opt/saltstack/salt/lib/python3.10/site-packages/requests/sessions.py", line 703, in send
                  r = adapter.send(request, **kwargs)
                File "/opt/saltstack/salt/lib/python3.10/site-packages/requests/adapters.py", line 637, in send
                  raise InvalidURL(e, request=request)
              requests.exceptions.InvalidURL: Not supported URL scheme http+docker
              
              The above exception was the direct cause of the following exception:
              
              Traceback (most recent call last):
                File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/state.py", line 2469, in call
                  ret = self.states[cdata["full"]](
                File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/loader/lazy.py", line 160, in __call__
                  ret = self.loader.run(run_func, *args, **kwargs)
                File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/loader/lazy.py", line 1269, in run
                  return self._last_context.run(self._run_as, _func_or_method, *args, **kwargs)
                File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/loader/lazy.py", line 1284, in _run_as
                  ret = _func_or_method(*args, **kwargs)
                File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/loader/lazy.py", line 1320, in wrapper
                  return f(*args, **kwargs)
                File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/utils/decorators/extension_deprecation.py", line 39, in wrapper
                  return function(*args, **salt.utils.args.clean_kwargs(**kwargs))
                File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/states/docker_container.py", line 1689, in running
                  image_id = _resolve_image(ret, image, client_timeout)
                File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/states/docker_container.py", line 190, in _resolve_image
                  image_id = __salt__["docker.resolve_image_id"](image)
                File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/loader/lazy.py", line 160, in __call__
                  ret = self.loader.run(run_func, *args, **kwargs)
                File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/loader/lazy.py", line 1269, in run
                  return self._last_context.run(self._run_as, _func_or_method, *args, **kwargs)
                File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/loader/lazy.py", line 1284, in _run_as
                  ret = _func_or_method(*args, **kwargs)
                File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/utils/decorators/extension_deprecation.py", line 39, in wrapper
                  return function(*args, **salt.utils.args.clean_kwargs(**kwargs))
                File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/modules/dockermod.py", line 2062, in resolve_image_id
                  inspect_result = inspect_image(name)
                File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/modules/dockermod.py", line 1994, in inspect_image
                  ret = _client_wrapper("inspect_image", name)
                File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/modules/dockermod.py", line 456, in wrapper
                  __context__["docker.client"] = _get_client(timeout=timeout, **kwargs)
                File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/modules/dockermod.py", line 416, in _get_client
                  ret = docker.APIClient(**client_kwargs)
                File "/opt/saltstack/salt/extras-3.10/docker/api/client.py", line 197, in __init__
                  self._version = self._retrieve_server_version()
                File "/opt/saltstack/salt/extras-3.10/docker/api/client.py", line 220, in _retrieve_server_version
                  raise DockerException(
              docker.errors.DockerException: Error while fetching server API version: Not supported URL scheme http+docker
     Started: 11:05:22.164102
    Duration: 2.938 ms
     Changes:   

Summary for local
------------
Succeeded: 0
Failed:    2
------------
Total states run:     2
Total run time:   7.773 ms

Versions Report

Salt Version:
          Salt: 3007.2
 
Python Version:
        Python: 3.10.17 (main, Apr 14 2025, 21:37:11) [GCC 11.2.0]
 
Dependency Versions:
          cffi: 1.16.0
      cherrypy: 18.8.0
  cryptography: 42.0.5
      dateutil: 2.8.2
     docker-py: 7.0.0
         gitdb: Not Installed
     gitpython: Not Installed
        Jinja2: 3.1.6
       libgit2: Not Installed
  looseversion: 1.3.0
      M2Crypto: 0.38.0
          Mako: Not Installed
       msgpack: 1.0.7
  msgpack-pure: Not Installed
  mysql-python: Not Installed
     packaging: 24.0
     pycparser: 2.21
      pycrypto: Not Installed
  pycryptodome: 3.19.1
        pygit2: Not Installed
  python-gnupg: 0.5.2
        PyYAML: 6.0.1
         PyZMQ: 25.1.2
        relenv: 0.19.0
         smmap: Not Installed
       timelib: 0.3.0
       Tornado: 6.4.2
           ZMQ: 4.3.4
 
Salt Extensions:
 saltext.mysql: 1.1.0
 
Salt Package Information:
  Package Type: onedir
 
System Versions:
          dist: debian 12.11 bookworm
        locale: utf-8
       machine: x86_64
       release: 6.8.12-5-pve
        system: Linux
       version: Debian GNU/Linux 12.11 bookworm

darkpixel avatar May 30 '25 18:05 darkpixel

https://github.com/docker/docker-py/issues/3256

you can try reverting the requests to an earlier version. however you should know that it was most likely updated because of security

you can try updating docker to 7.1.0 but i don't know if that version works with salt. i have never tried it.

and finally it should be noted that until someone takes over the docker modules in a saltext that it most likely will not be updated and will be dropped in 3008.

whytewolf avatar Jun 02 '25 15:06 whytewolf

It doesn't work with 7.1.0 either, but nice touch with the deprecation warnings.

# salt-call -l info pip.install docker==7.1.0
# salt-call --versions-report
Salt Version:
          Salt: 3007.3
 
Python Version:
        Python: 3.10.17 (main, May 11 2025, 04:07:13) [GCC 11.2.0]
 
Dependency Versions:
          cffi: 1.16.0
      cherrypy: 18.8.0
  cryptography: 42.0.5
      dateutil: 2.8.2
     docker-py: 7.1.0
/snip

# salt-call state.sls dockertest
          ID: helloworld
    Function: docker_image.present
        Name: helloworld
      Result: False
     Comment: An exception occurred in this state: Traceback (most recent call last):
                File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/state.py", line 2469, in call
                  ret = self.states[cdata["full"]](
                File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/loader/lazy.py", line 160, in __call__
                  ret = self.loader.run(run_func, *args, **kwargs)
                File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/loader/lazy.py", line 1269, in run
                  return self._last_context.run(self._run_as, _func_or_method, *args, **kwargs)
                File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/loader/lazy.py", line 1284, in _run_as
                  ret = _func_or_method(*args, **kwargs)
                File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/loader/lazy.py", line 1320, in wrapper
                  return f(*args, **kwargs)
                File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/utils/decorators/extension_deprecation.py", line 39, in wrapper
                  return function(*args, **salt.utils.args.clean_kwargs(**kwargs))
                File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/states/docker_image.py", line 254, in present
                  image_info = __salt__["docker.inspect_image"](full_image)
                File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/loader/lazy.py", line 160, in __call__
                  ret = self.loader.run(run_func, *args, **kwargs)
                File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/loader/lazy.py", line 1269, in run
                  return self._last_context.run(self._run_as, _func_or_method, *args, **kwargs)
                File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/loader/lazy.py", line 1284, in _run_as
                  ret = _func_or_method(*args, **kwargs)
                File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/utils/decorators/extension_deprecation.py", line 39, in wrapper
                  return function(*args, **salt.utils.args.clean_kwargs(**kwargs))
                File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/modules/dockermod.py", line 1994, in inspect_image
                  ret = _client_wrapper("inspect_image", name)
                File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/modules/dockermod.py", line 456, in wrapper
                  __context__["docker.client"] = _get_client(timeout=timeout, **kwargs)
                File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/modules/dockermod.py", line 416, in _get_client
                  ret = docker.APIClient(**client_kwargs)
                File "/opt/saltstack/salt/extras-3.10/docker/api/client.py", line 141, in __init__
                  self._auth_configs = auth.load_config(
              TypeError: load_config() got an unexpected keyword argument 'config_dict'
     Started: 13:28:46.413012
    Duration: 3.945 ms
     Changes:   

Maybe the release notes should be updated so other people don't get 250 machines deep before running into this. ;)

darkpixel avatar Jun 06 '25 20:06 darkpixel

they have been more then vocal about what is getting dropped in 3008. everything in this list https://github.com/saltstack/great-module-migration/blob/main/community-ext-modules.txt is getting dropped in 3008. and it is expected of the community to pick them up with they want those modules. a large portion of them have been picked up in https://github.com/salt-extensions

the migration became vital to get the project manageable by a smaller team. so the standard deprecation warnings have not been followed for this.

docker has not been picked up yet. guess not enough people are using it to warrant it needing a community edition.

whytewolf avatar Jun 06 '25 22:06 whytewolf

Totally understood--but this is broken before 3008.x.

While I can't magically fix the number of people on your team, maybe updating the release notes to say "This version breaks docker and we're not going to fix it unless the community steps up"?

If not, no worries.

darkpixel avatar Jun 08 '25 21:06 darkpixel

I'm not on the core team. so it isn't my team. I'm just another user.

whytewolf avatar Jun 08 '25 22:06 whytewolf

@darkpixel I no longer have the error and everything works correctly again for me, after upgrade to Python lib docker=7.1.0 and salt-minion 3007.5.

I use Debian 12 (with salt-minion DEB packages from Saltstack repository) with "package-formula" to deploy the Python lib for Docker.

packages:
  pips:
    required:
      pkgs:
        - python3-pip

    wanted:
      - docker

My Docker version :

docker --version
Docker version 28.3.1, build 38b7060

My Salt Versions Report :

salt-call --versions-report
Salt Version:
          Salt: 3007.5
 
Python Version:
        Python: 3.10.17 (main, Jun  9 2025, 20:41:48) [GCC 11.2.0]
 
Dependency Versions:
          cffi: 1.16.0
      cherrypy: 18.6.1
  cryptography: 42.0.5
      dateutil: 2.8.2
     docker-py: 7.1.0
         gitdb: Not Installed
     gitpython: Not Installed
        Jinja2: 3.1.6
       libgit2: Not Installed
  looseversion: 1.3.0
      M2Crypto: Not Installed
          Mako: Not Installed
       msgpack: 1.0.7
  msgpack-pure: Not Installed
  mysql-python: Not Installed
     packaging: 24.0
     pycparser: 2.21
      pycrypto: Not Installed
  pycryptodome: 3.19.1
        pygit2: Not Installed
  python-gnupg: 0.5.2
        PyYAML: 6.0.1
         PyZMQ: 25.1.2
        relenv: 0.19.3
         smmap: Not Installed
       timelib: 0.3.0
       Tornado: 6.4.2
           ZMQ: 4.3.4
 
Salt Package Information:
  Package Type: onedir
 
System Versions:
          dist: debian 12.11 bookworm
        locale: utf-8
       machine: x86_64
       release: 6.8.12-11-pve
        system: Linux
       version: Debian GNU/Linux 12.11 bookworm`

I hope this can help you.

Regards,

kartnico avatar Jul 06 '25 13:07 kartnico

Apparently it's a conflicting docker library somewhere under /opt/saltstack.

apt-get autoremove --purge salt-minion salt-common
rm -rf /opt/saltstack
apt-get install salt-minion
salt-call -l info pip.install docker

darkpixel avatar Aug 11 '25 20:08 darkpixel