aws
Instructions for Setting up ECS Agent Container (i.e. Other Linux AMIs when awsvpc networking mode is enabled) Do Not Work
Summary
I am unable to set up the ECS Agent container using the instruction set forth in https://github.com/aws/amazon-ecs-agent#on-other-linux-amis-when-awsvpc-networking-mode-is-enabled.
Description
A bit of context, I want to create an Ubuntu AMI with the AWS ECS Agent installed so that I can use https://github.com/nestybox/sysbox as the default container runtime. Related open issue is here: https://github.com/aws/containers-roadmap/issues/673. For this reason, I am looking to use the following instructions to manually set up the ECS agent using a docker run command so that I can also use the --runtime flag to specify the ECS Agent to use runc while I set the default container runtime for the other containers to be sysbox-runc:
mkdir -p /var/log/ecs /etc/ecs /var/lib/ecs/data
touch /etc/ecs/ecs.config
echo ECS_CLUSTER=<cluster-id> >> /etc/ecs/ecs.config
echo ECS_ENABLE_AWSLOGS_EXECUTIONROLE_OVERRIDE=true >> /etc/ecs/ecs.config
sysctl -w net.ipv4.conf.all.route_localnet=1
iptables -t nat -A PREROUTING -p tcp -d 169.254.170.2 --dport 80 -j DNAT --to-destination 127.0.0.1:51679
iptables -t nat -A OUTPUT -d 169.254.170.2 -p tcp -m tcp --dport 80 -j REDIRECT --to-ports 51679
/usr/bin/docker run --name ecs-agent \
--init \
--restart=on-failure:10 \
--volume=/var/run:/var/run \
--volume=/var/log/ecs/:/log:Z \
--volume=/var/lib/ecs/data:/data:Z \
--volume=/etc/ecs:/etc/ecs \
--volume=/sbin:/host/sbin \
--volume=/lib:/lib \
--volume=/lib64:/lib64 \
--volume=/usr/lib:/usr/lib \
--volume=/usr/lib64:/usr/lib64 \
--volume=/proc:/host/proc \
--volume=/sys/fs/cgroup:/sys/fs/cgroup \
--net=host \
--env-file=/etc/ecs/ecs.config \
--cap-add=sys_admin \
--cap-add=net_admin \
--env ECS_ENABLE_TASK_ENI=true \
--env ECS_UPDATES_ENABLED=true \
--env ECS_ENGINE_TASK_CLEANUP_WAIT_DURATION=1h \
--env ECS_DATADIR=/data \
--env ECS_ENABLE_TASK_IAM_ROLE=true \
--env ECS_ENABLE_TASK_IAM_ROLE_NETWORK_HOST=true \
--env ECS_LOGFILE=/log/ecs-agent.log \
--env ECS_AVAILABLE_LOGGING_DRIVERS='["json-file","awslogs","syslog","none"]' \
--env ECS_LOGLEVEL=info \
--detach \
amazon/amazon-ecs-agent:latest
Using the above setup instructions, I can get the agent to connect to the ECS cluster; however, I cannot get my tasks to run and get the following error:
The closest matching container-instance <container-arn> is missing an attribute required by your task. For more information, see the [Troubleshooting section](http://docs.aws.amazon.com/AmazonECS/latest/developerguide/troubleshooting.html).
I've already cross-referenced the required task definition attributes vs. the container instance attributes and they are matching:
Required Task Definition Attributes
"requiresAttributes": [
{
"targetId": null,
"targetType": null,
"value": null,
"name": "com.amazonaws.ecs.capability.logging-driver.awslogs"
},
{
"targetId": null,
"targetType": null,
"value": null,
"name": "ecs.capability.execution-role-awslogs"
},
{
"targetId": null,
"targetType": null,
"value": null,
"name": "com.amazonaws.ecs.capability.ecr-auth"
},
{
"targetId": null,
"targetType": null,
"value": null,
"name": "com.amazonaws.ecs.capability.docker-remote-api.1.19"
},
{
"targetId": null,
"targetType": null,
"value": null,
"name": "com.amazonaws.ecs.capability.task-iam-role"
},
{
"targetId": null,
"targetType": null,
"value": null,
"name": "ecs.capability.execution-role-ecr-pull"
},
{
"targetId": null,
"targetType": null,
"value": null,
"name": "com.amazonaws.ecs.capability.docker-remote-api.1.18"
},
{
"targetId": null,
"targetType": null,
"value": null,
"name": "ecs.capability.task-eni"
}
],
Container Instance Attributes
"attributes": [
{
"name": "ecs.capability.secrets.asm.environment-variables"
},
{
"name": "ecs.capability.branch-cni-plugin-version",
"value": "199bfc6-1.2-5-g199bfc6"
},
{
"name": "ecs.ami-id",
"value": "ami-08d4ac5b634553e16"
},
{
"name": "ecs.capability.secrets.asm.bootstrap.log-driver"
},
{
"name": "ecs.capability.task-eia.optimized-cpu"
},
{
"name": "com.amazonaws.ecs.capability.logging-driver.none"
},
{
"name": "ecs.capability.ecr-endpoint"
},
{
"name": "ecs.capability.docker-plugin.local"
},
{
"name": "ecs.capability.task-cpu-mem-limit"
},
{
"name": "ecs.capability.secrets.ssm.bootstrap.log-driver"
},
{
"name": "com.amazonaws.ecs.capability.docker-remote-api.1.30"
},
{
"name": "ecs.capability.full-sync"
},
{
"name": "com.amazonaws.ecs.capability.docker-remote-api.1.31"
},
{
"name": "com.amazonaws.ecs.capability.docker-remote-api.1.32"
},
{
"name": "ecs.capability.firelens.options.config.file"
},
{
"name": "ecs.capability.logging-driver.awsfirelens.log-driver-buffer-limit"
},
{
"name": "ecs.availability-zone",
"value": "us-east-1d"
},
{
"name": "ecs.capability.aws-appmesh"
},
{
"name": "com.amazonaws.ecs.capability.logging-driver.awslogs"
},
{
"name": "com.amazonaws.ecs.capability.docker-remote-api.1.24"
},
{
"name": "com.amazonaws.ecs.capability.docker-remote-api.1.25"
},
{
"name": "ecs.capability.task-eni-trunking"
},
{
"name": "com.amazonaws.ecs.capability.docker-remote-api.1.26"
},
{
"name": "com.amazonaws.ecs.capability.docker-remote-api.1.27"
},
{
"name": "com.amazonaws.ecs.capability.docker-remote-api.1.28"
},
{
"name": "com.amazonaws.ecs.capability.privileged-container"
},
{
"name": "com.amazonaws.ecs.capability.docker-remote-api.1.29"
},
{
"name": "ecs.cpu-architecture",
"value": "x86_64"
},
{
"name": "com.amazonaws.ecs.capability.ecr-auth"
},
{
"name": "ecs.capability.firelens.fluentbit"
},
{
"name": "com.amazonaws.ecs.capability.docker-remote-api.1.20"
},
{
"name": "ecs.os-type",
"value": "linux"
},
{
"name": "com.amazonaws.ecs.capability.docker-remote-api.1.21"
},
{
"name": "com.amazonaws.ecs.capability.docker-remote-api.1.22"
},
{
"name": "ecs.capability.task-eia"
},
{
"name": "com.amazonaws.ecs.capability.docker-remote-api.1.23"
},
{
"name": "ecs.capability.private-registry-authentication.secretsmanager"
},
{
"name": "com.amazonaws.ecs.capability.logging-driver.syslog"
},
{
"name": "com.amazonaws.ecs.capability.logging-driver.awsfirelens"
},
{
"name": "ecs.capability.firelens.options.config.s3"
},
{
"name": "com.amazonaws.ecs.capability.logging-driver.json-file"
},
{
"name": "ecs.capability.execution-role-awslogs"
},
{
"name": "ecs.vpc-id",
"value": "vpc-093e498597bb32b94"
},
{
"name": "com.amazonaws.ecs.capability.docker-remote-api.1.17"
},
{
"name": "com.amazonaws.ecs.capability.docker-remote-api.1.18"
},
{
"name": "com.amazonaws.ecs.capability.docker-remote-api.1.19"
},
{
"name": "ecs.capability.task-eni"
},
{
"name": "ecs.capability.firelens.fluentd"
},
{
"name": "ecs.capability.efs"
},
{
"name": "ecs.capability.execution-role-ecr-pull"
},
{
"name": "ecs.capability.task-eni.ipv6"
},
{
"name": "ecs.capability.container-health-check"
},
{
"name": "ecs.os-family",
"value": "LINUX"
},
{
"name": "ecs.subnet-id",
"value": "subnet-07e96d0adeeb4ce8d"
},
{
"name": "ecs.instance-type",
"value": "t2.micro"
},
{
"name": "com.amazonaws.ecs.capability.task-iam-role-network-host"
},
{
"name": "ecs.capability.increased-task-cpu-limit"
},
{
"name": "ecs.capability.container-ordering"
},
{
"name": "ecs.capability.cni-plugin-version",
"value": "db586472-2020.09.0"
},
{
"name": "ecs.capability.env-files.s3"
},
{
"name": "ecs.capability.secrets.ssm.environment-variables"
},
{
"name": "ecs.capability.pid-ipc-namespace-sharing"
},
{
"name": "com.amazonaws.ecs.capability.task-iam-role"
}
],
Even running the following commands shows that there are no missing attributes:
(base) andrew@Andrews-MacBook-Pro-2 Desktop % ecs-cli check-attributes --cluster <cluster-id> --task-def <task-id> --container-instances <container-id>
Container Instance Missing Attributes
<cluster-name> None
Looking into the docs further (https://aws.amazon.com/premiumsupport/knowledge-center/ecs-container-instance-requirement-error/), it seems that it could be that my ecs cluster and container instance may not be on the same subnet (as I'm using the awsvpc networking mode for my ECS cluster), but after confirming that they are on the same subnet, I still get the same issue.
What's strange is when I use the ECS-optimized image https://us-east-1.console.aws.amazon.com/systems-manager/parameters/aws/service/ecs/optimized-ami/amazon-linux-2/recommended/image_id/description?region=us-east-1# and use systemctl start ecs, my ECS agent works fine and tasks are picked up successfully, but when I try to manually invoke docker run to start the ECS agent using the instructions above (on the same ECS-optimized AMI), I get the same required attributes error.
Any thoughts? I've been stuck on this for a while :(
Expected Behavior
The ECS Agent should be able to connect to the ECS cluster and pick up tasks.
Observed Behavior
The ECS Agent connects to my ECS cluster but fails to pick up tasks due to an error that my container instance is missing an attribute required by your task
Environment Details
I'm attempting to set up an Ubuntu ECS AMI using the AMI https://aws.amazon.com/marketplace/pp/prodview-iftkyuwv2sjxi?sr=0-2&ref_=beagle&applicationId=AWSMPContessa Ran a cross-test on the ECS-optimized AMI for https://us-east-1.console.aws.amazon.com/systems-manager/parameters/aws/service/ecs/optimized-ami/amazon-linux-2/recommended/image_id/description?region=us-east-1#
