In ECS context: docker compose convert/up API throttling problems

[thorfi]

trafficstars

Description

docker compose up and docker compose convert fail due to AWS API throttling if there are over 100 EFS file systems.

Also docker compose up fails due to AWS API throttling while waiting for the stack to complete on large stacks with a lot of resources as the number of describes goes over the thresholds.

In both cases, AWS API documentation recommends using exponential backoff to allow the API call count bucket to refill, rather than just crashing.

https://docs.aws.amazon.com/AWSEC2/latest/APIReference/query-api-troubleshooting.html#api-request-rate

( Re-raised from docker/compose#9177 )

Steps to reproduce the issue:

1. Have a docker-compose.yaml that:
   • involves at least one volume entry and >= 100 EFS volumes that already exist OR
   • contains >= 100 Resources that docker compose monitors "live" during docker compose up
2. docker context use ecs-context
3. docker compose --verbose up or docker compose --verbose convert

Describe the results you received:

ThrottlingException: status code: 400, request id: 2259a3c2-1130-4034-a41f-24914d8fa550

Describe the results you expected:

docker compose up - CloudFormation Stack should be created docker compose convert - CloudFormation .yaml code should be printed

Additional information you deem important (e.g. issue happens only occasionally):

This was not happening in docker compose convert previously, but now it is. docker compose up has had the problem for a long time.

Having done some digging with --verbose on:

if there are < 100 EFS entries everything works:

...
DEBUG Retrieve default VPC
DEBUG Retrieve SubNets
DEBUG searching for existing filesystem as volume "opt_o3c_server_ca"
DEBUG searching for existing filesystem as volume "etc_ssh_mount"
DEBUG Create CloudFormation stack
DEBUG Stack arn:aws:cloudformation:ap-southeast-2:657831033816:stack/o3c-17087db/afae9c30-8edf-11ec-8bb5-02b1b681ea46 created
...

If there are >= 100 EFS ids it fails:

DEBUG Retrieve default VPC
DEBUG Retrieve SubNets
DEBUG searching for existing filesystem as volume "etc_ssh_mount"
ThrottlingException:
	status code: 400, request id: 128818eb-59b6-4303-96d7-a449a6c5c929

Related AWS Documentation:

• https://docs.aws.amazon.com/AWSEC2/latest/APIReference/throttling.html#throttling-limits This says there's a limit of 100 Get or Describe operations, which seems to match the threshold.
• https://docs.aws.amazon.com/AWSEC2/latest/APIReference/query-api-troubleshooting.html#api-request-rate Recommendation for exponential backoff in the case of throttling errors

Output of docker compose version:

% docker context use ecs-context
Docker Compose version dev
% docker context use default
Docker Compose version v2.2.3

Output of docker info:

Client:
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc., v0.7.1)
  compose: Docker Compose (Docker Inc., v2.2.3)
  scan: Docker Scan (Docker Inc., v0.17.0)

Server:
 Containers: 1
  Running: 1
  Paused: 0
  Stopped: 0
 Images: 14
 Server Version: 20.10.12
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 7b11cfaabd73bb80907dd23182b9347b4245eb5d
 runc version: v1.0.2-0-g52b36a2
 init version: de40ad0
 Security Options:
  seccomp
   Profile: default
  cgroupns
 Kernel Version: 5.10.76-linuxkit
 Operating System: Docker Desktop
 OSType: linux
 Architecture: aarch64
 CPUs: 4
 Total Memory: 2.423GiB
 Name: docker-desktop
 ID: P5SC:JGRV:E5O6:N6SH:7TCP:HA6N:XDIS:A3VT:KMQL:XAVX:H23E:AGPR
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 HTTP Proxy: http.docker.internal:3128
 HTTPS Proxy: http.docker.internal:3128
 No Proxy: hubproxy.docker.internal
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  hubproxy.docker.internal:5000
  127.0.0.0/8
 Live Restore Enabled: false

Additional environment details:

macOS Monterey, MacBook Pro M1.

[thorfi]

bump

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[stale[bot]]

This issue has been automatically closed because it had not recent activity during the stale period.