Support docker with minAPIVerion > 1.21

[pmv]

Summary

ECS agent should support minAPIVerson > 1.21

Description

I am attempting to set up an ECS anywhere cluster with RHEL and podman. I realize ECS anywhere lists docker as the requirement, but since podman claims to be a mostly drop-in replacement for docker and it's what RHEL supports I thought I'd try it.

What I've found is amazon-ecs-agent fails to start due to the version detection logic / default version. This is because podman included with RHEL 8.3 reports the apiVersion as 1.40 and the minAPIVersion as 1.24. From the agent log:

level=critical time=2021-06-29T21:29:49Z msg="Error creating Docker client: docker client factory: client not found for docker version: 1.21" module=agent.go

I was able to get by the error by changing this https://github.com/aws/amazon-ecs-agent/blob/v1.53.1/agent/app/agent.go#L769-L774 to accept versions > 1.21 instead of looking for only 1.21 (since the minimum version is 1.24, the 1.21 check failed).

I also had to change this: https://github.com/aws/amazon-ecs-agent/blob/v1.53.1/agent/dockerclient/sdkclientfactory/versionsupport_unix.go#L56 so it would fall within the range of acceptable versions for my setup (>= 1.24 and < 1.40).

I understand hard-coding a higher default version is likely not feasible - perhaps better logic would be

1. if minAPIVersion <= 1.21, the default version = 1.21
2. if minAPIVersion > 1.21, make the default version = minAPIVersion ?

[yinyic]

Hi @pmv , thank you for raising this issue to our attention.

We created PR aws/amazon-ecs-agent#2937 , but have decided to hold it off after discussing with team. This is because we found that more profound changes with other ECS components need to happen for supporting this change properly, otherwise we may see undesired instance placement limitations.

We have added backlog item to track this issue, and will update this thread once we get to work on it :)

[sharanyad]

Since this would involve changes other than just the ECS agent to support this, I'm moving this over to containers-roadmap

[frehov]

Is there any movement on this? @yinyic @sharanyad

from the release of docker engine 25 they have deprecated older api versions, and you have to explicitly set an override in the docker service to allow older clients. https://docs.docker.com/engine/deprecated/#deprecate-legacy-api-versions

Use of old API versions is very rare, and support for legacy API versions involves significant complexity (Docker 1.0.0 having been released 10 years ago). Because of this, we'll start deprecating support for legacy API versions.

Docker Engine v25.0 by default disables API version older than 1.24 (aligning the minimum supported API version between Linux and Windows daemons).

Moving forward with the release of docker engine v26 the following will happen.

Support for API versions lower than 1.24 will be permanently removed in Docker Engine v26, and the minimum supported API version will be incrementally raised in releases following that.

This will undoubtedly break a lot of ecs-anywhere installations that may have automated updates of the ecs-agent and docker versions through SSM or other means.

This seems like it should get some attention, since automated installs with ecs-anywhere-install.sh would start failing silently(?) as it would install an up to date version of docker that has deprecated the client api ecs-agent is using.

[frehov]

this should probably be close due to https://github.com/aws/containers-roadmap/issues/1437 and it being release in version 1.81.0 of the ecs-agent.

We do however see issues with this, because of this issue that is currently breaking our installs https://github.com/aws/amazon-ecs-agent/issues/4089