Ansible / docker==7.0.0 docker compose up -> kwargs_from_env ssl_version

[Benouare]

Hello

Env : Debian 12, python3.9.2, Docker version 24.0.7, build afdd53b

With fresh pip install docker==7.0.0

- name: deploy docker
  become: true
  community.docker.docker_compose:
    project_src: /opt/config/docker/
    files:
    - docker-compose.yml

My task is failing with this error :

TASK [deploy docker] *********************************************************************************************************
fatal: [install_nas]: FAILED! => {"changed": false, "msg": "Configuration error - kwargs_from_env() got an unexpected keyword argument 'ssl_version'"}

With the 6.1.3 it's ok.

It looks like if i run it manually (directly on the target host) with (pip) docker===7.0.0 it's working....

Cheers !

[rvalle]

also experiencing this, docker-compose seems broken with the latest version of the docker python module. I am on Python 3.9 and Debian 11 if that helps.

[aapjeisbaas]

Same here on: ubuntu 20.04 AMD64 python: 3.10.12 docker-compose 1.29.2

docker 6.1.3 works docker 7.0.0 broken

ansible -vvv error

The full traceback is:
  File "/tmp/ansible_community.docker.docker_compose_payload_87nbuv_v/ansible_community.docker.docker_compose_payload.zip/ansible_collections/community/docker/plugins/modules/docker_compose.py", line 727, in __init__
  File "/usr/local/lib/python3.10/dist-packages/compose/cli/command.py", line 60, in project_from_options
    return get_project(
  File "/usr/local/lib/python3.10/dist-packages/compose/cli/command.py", line 152, in get_project
    client = get_client(
  File "/usr/local/lib/python3.10/dist-packages/compose/cli/docker_client.py", line 41, in get_client
    client = docker_client(
  File "/usr/local/lib/python3.10/dist-packages/compose/cli/docker_client.py", line 124, in docker_client
    kwargs = kwargs_from_env(environment=environment, ssl_version=tls_version)
fatal: [network-installer]: FAILED! => {
    "changed": false,
    "invocation": {
        "module_args": {
            "api_version": "auto",
            "build": false,
            "ca_cert": null,
            "client_cert": null,
            "client_key": null,
            "debug": false,
            "definition": null,
            "dependencies": true,
            "docker_host": "unix://var/run/docker.sock",
            "env_file": null,
            "files": null,
            "hostname_check": false,
            "nocache": false,
            "profiles": null,
            "project_name": null,
            "project_src": "/etc/compose/",
            "pull": false,
            "recreate": "smart",
            "remove_images": null,
            "remove_orphans": false,
            "remove_volumes": false,
            "restarted": false,
            "scale": null,
            "services": null,
            "ssl_version": null,
            "state": "present",
            "stopped": false,
            "timeout": null,
            "tls": false,
            "tls_hostname": null,
            "use_ssh_client": false,
            "validate_certs": false
        }
    },
    "msg": "Configuration error - kwargs_from_env() got an unexpected keyword argument 'ssl_version'"
}

[milas]

The ssl_version argument was removed in 7.0.0. All supported versions of Python now have native TLSv1.3 support and no longer require this.

Please consider pinning at least by major version to avoid surprise breakage like this.

[milas]

See also: upgrade notes in the release @ https://github.com/docker/docker-py/releases/tag/7.0.0

[marcsc13]

In compose/cli/docker_client.py on line 124 the call of kwargs_from_env expects the function signature to be kwargs_from_env(environment=None, ssl_version=None) but it was changed to

def kwargs_from_env(environment=None):

(See file docker/utils/utils.py line 344)

[milas]

In compose/cli/docker_client.py on line 124

Compose v1 (the Python version) is no longer maintained or supported.

Compose v2 is bundled with Docker Desktop, included in the apt/yum repos for Docker Engine on Linux, available standalone on GitHub @ https://github.com/docker/compose/releases, etc.

[marcsc13]

Do I understand you correctly that compose/cli/docker_client.py refers to the Compose v1 instead of the current Compose v2 API? Should I report the deprecated API usage to the community maintainers of community.docker.docker_compose then?

I am confused since the change in release 7.0.0 (https://github.com/docker/docker-py/compare/6.1.3...7.0.0#diff-c40bbe508dae196428f3740f4aa8fbd143f8e3c0d80aa3e6952938cb2f90addbL332) changed the kwargs_from_env signature which makes ansibles community.docker.docker_compose break when using it like @Benouare described.

Here is a basic example of mine:

- name: Deploy traefik
  community.docker.docker_compose: 
    project_src: "{{ project_directory }}"
    state: present
  become: true

[felixfontein]

compose v2 is written in Go, not in Python. compose v1 does not work with Docker SDK for Python 7.0.0+, only with versions up to 6.x.y. You need to pin the version of Docker SDK for Python to avoid installing an invalid combination of the no longer maintained docker-compose v1 and Docker SDK for Python.

Should I report the deprecated API usage to the community maintainers of community.docker.docker_compose then?

See https://github.com/ansible-collections/community.docker/issues/216.

[nishantvarma]

@marcsc13 , You are right, compose/cli/docker_client.py refers to v1; however, that's also released by Docker, so your real option seems to be to either downgrade Docker's Python wrapper: pip install docker==6.1.3 (temporary workaround) OR remove docker-compose and start using the inbuilt docker compose (sub-command). I tried the latter, but that's breaking some of my YAML files, so resorting to 1 right now.

[sgreinerCNS]

docker compose (v2) is not available on many current LTS Linux distributions, e.g. Ubuntu Server 22.04. It only has docker-compose (v1), docker compose (v2) is not available without installing docker from third party repos. So I think forcing the world to v2 cannot be a solution here.

If docker-compose v1.29.2 python lib is not compatible with docker-py 7.0.0 this should be reflected in its dependencies (docker<7.0.0)

But honestly, docker-py 7.x should simply not make changes which breaks the old docker-compose v1.29.2

We had to pin dozens of systems now to docker-py 6.1.3 because also our ansible based deployments were broken. Luckily we noticed that bug on a test system before any live-servers got affected. Ansible (which is very wildly used for docker compose deployments) itself still does not have a stable compose v2 implementation.

[SonGokussj4]

Same problem... Very unpleasant experience... Had to degrade to 6.1.3

[goldyfruit]

Same problem for me as well.

Maybe a "compat" function with deprecation message would have been a smoother transition.

[oulfr]

to resolve this issue you need to downgrade the version of docker:

     pip uninstall docker
     pip install docker==6.1.3

[pgassmann]

Docker compatibility is fixed in ansible collection community.docker Version 3.5.0 https://github.com/ansible-collections/community.docker/blob/main/CHANGELOG.rst#v3-5-0

It is currently not part of the ansible-community release: https://github.com/ansible-community/ansible-build-data/blob/main/9/CHANGELOG-v9.rst

You can install the new version of the collection manually: ansible-galaxy collection install community.docker --force

[pgassmann]

unfortunatly not yet a complete solution.

community.docker.docker_compose still uses docker-compose pip library which is broken.

TASK [teamapps.general.webproxy : webproxy_docker-compose-up] **********************************************************************************************************************************************************
task path: /home/pgassmann/git/project-ansible/roles/webproxy/tasks/webproxy.yml:116
The full traceback is:
  File "/tmp/ansible_docker_compose_payload_8rb5fnzd/ansible_docker_compose_payload.zip/ansible_collections/community/docker/plugins/modules/docker_compose.py", line 727, in __init__
  File "/usr/local/lib/python3.10/dist-packages/compose/cli/command.py", line 60, in project_from_options
    return get_project(
  File "/usr/local/lib/python3.10/dist-packages/compose/cli/command.py", line 152, in get_project
    client = get_client(
  File "/usr/local/lib/python3.10/dist-packages/compose/cli/docker_client.py", line 41, in get_client
    client = docker_client(
  File "/usr/local/lib/python3.10/dist-packages/compose/cli/docker_client.py", line 124, in docker_client
    kwargs = kwargs_from_env(environment=environment, ssl_version=tls_version)
fatal: [website-dev1.example.com]: FAILED! => changed=false 
  invocation:
    module_args:
      api_version: auto
      build: false
      ca_cert: null
      client_cert: null
      client_key: null
      debug: false
      definition: null
      dependencies: true
      docker_host: unix://var/run/docker.sock
      env_file: null
      files: null
      hostname_check: false
      nocache: false
      profiles: null
      project_name: null
      project_src: /container/webproxy
      pull: false
      recreate: smart
      remove_images: null
      remove_orphans: true
      remove_volumes: false
      restarted: false
      scale: null
      services: null
      ssl_version: null
      state: present
      stopped: false
      timeout: null
      tls: false
      tls_hostname: null
      use_ssh_client: false
      validate_certs: false
  msg: Configuration error - kwargs_from_env() got an unexpected keyword argument 'ssl_version'

docker-compose command is broken. not sure if there will be a fixed python docker-compose version. https://pypi.org/project/docker-compose/

UPDATE: There won't be a new release of v1 of docker-compose. https://github.com/docker/compose/issues/11168

root@website-dev1 /container/webproxy # docker-compose up -d
Traceback (most recent call last):
  File "/usr/local/bin/docker-compose", line 8, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.10/dist-packages/compose/cli/main.py", line 81, in main
    command_func()
  File "/usr/local/lib/python3.10/dist-packages/compose/cli/main.py", line 200, in perform_command
    project = project_from_options('.', options)
  File "/usr/local/lib/python3.10/dist-packages/compose/cli/command.py", line 60, in project_from_options
    return get_project(
  File "/usr/local/lib/python3.10/dist-packages/compose/cli/command.py", line 152, in get_project
    client = get_client(
  File "/usr/local/lib/python3.10/dist-packages/compose/cli/docker_client.py", line 41, in get_client
    client = docker_client(
  File "/usr/local/lib/python3.10/dist-packages/compose/cli/docker_client.py", line 124, in docker_client
    kwargs = kwargs_from_env(environment=environment, ssl_version=tls_version)
TypeError: kwargs_from_env() got an unexpected keyword argument 'ssl_version'

There is some work on a new docker_compose_v2 ansible module. not released yet.

• https://github.com/ansible-collections/community.docker/pull/586
• https://ansible-collections.github.io/community.docker/pr/586/docker_compose_v2_module.html

so currently the only option is to downgrade the python docker library pip install docker==6.1.3 or to manually patch the docker-compose library.

[felixfontein]

The fix does not cover docker_compose nor claims it does. The only thing that changes for the docker_compose module is that its documentation now mentionst that you need Docker SDK for Python < 7.0.0 (https://docs.ansible.com/ansible/devel/collections/community/docker/docker_compose_module.html#requirements).

There is some work on a new docker_compose_v2 ansible module. not released yet.

If anyone wants to pick up that work, I'll happily help.

[agowa]

@pgassmann either it is broken again with version 3.6.0 of community.docker, or your suggestion doesn't work. I've pinned all of the collections into my project and it just broke after updating to the latest collection version for me too.

[felixfontein]

@agowa you need to restrict the version of Docker SDK for Python. That's the only way to fix this.

[agowa]

On the controller I've docker-compose==1.29.2 pinned within the requirements.txt. Apparently that isn't enough. Do I need to pin it on the target as well? That could be a bit annoying as (if it is installed at all, don't know atm) it's installed from the repositories, so it'll be a bit annoying to downgrade...

Edit: Oh, docker SDK is docker not docker-compose, so I've to add docker==6.1.3, ok.

[pgassmann]

Update: In the lastest ansible release 9.2.0, the new docker_compose_v2 modules are available. https://docs.ansible.com/ansible/latest/collections/community/docker/docker_compose_v2_module.html

there's a separate module for more specific docker compose pull tasks: https://docs.ansible.com/ansible/latest/collections/community/docker/docker_compose_v2_pull_module.html#ansible-collections-community-docker-docker-compose-v2-pull-module

[nishantvarma]

Docker Compose v2 is worth the switch, btw.

[collinsoden]

I encountered this when I installed docker with pip on a new device and also when I installed with apt, I had to reinstall the snap version of docker with

sudo snap install docker

This fixed everything and I was able to run: docker compose up again

[ndcast]

Easy fix for my ansible set up was

sudo vim /usr/bin/docker-compose then

#!/bin/bash docker compose $1 $2 $3 $4 solved.

[felixfontein]

@ndcast that will very likely not help if you use Ansible's community.docker.docker_compose module, since that module uses docker-compose as a Python library and doesn't care about the contents of the docker-compose binary.

You should better use the community.docker.docker_compose_v2 module (see @pgassmann's post above), which directly uses docker compose.

Also besides that, your wrapper is only passing the first four parameters, and it's not quoting them correctly. If you really want to use such a wrapper, better use this:

#!/bin/bash
docker compose "$@"

(Ref: https://www.gnu.org/software/bash/manual/bash.html#index-_0040)