aws
ECS autoscaling with graceful shutdown issue
I have an ECS (EC2 launch type) setup running a PHP worker. The ECS service scales based on an SQS queue, with desired = 1, min = 1, max = 2. The Auto Scaling Group (ASG) has min = 1, desired = 1, max = 3. The ECS task performs a graceful shutdown that can take up to 1 hour (the EC2 instance itself stays running). The issue I’m seeing is: When a scale-out event happens during that graceful shutdown, ECS keeps trying to place the new task on the same EC2 instance where the old task is shutting down — even though that instance has no remaining capacity, and there are available unprovisioned instances in the ASG. I already use the distinctInstance placement constraint, but the scheduler still attempts placement on the busy instance. Has anyone seen this before or found a way to prevent ECS from scheduling new tasks on instances that are hosting tasks in graceful shutdown (without draining the instance itself)? Any guidance or best practices would be really appreciated
Hi @naveen2112 , Could you please drop an email to [email protected] linking this issue and we'd be happy to help you out? Please include details about your use case and configuration as well.
Hello @naveen2112! Thanks for opening this issue. Currently, this is by design as we follow an "optimistic task resource accounting" when placing tasks. To try to modify how tasks are placed, you could look to utilize some of the task placement strategies that ECS supports for ECS services (e.g. binpack, random, spread). -> https://docs.aws.amazon.com/AmazonECS/latest/developerguide/task-placement-strategies.html
If none of these work for you, please feel free to cut a support ticket over to us and/or raise a request in containers-roadmap describing what you would like ECS to support.
Other refs:
- https://aws.amazon.com/blogs/containers/improvements-to-amazon-ecs-task-launch-behavior-when-tasks-have-prolonged-shutdown/
- https://aws.amazon.com/about-aws/whats-new/2023/06/amazon-ecs-tasks-faster-prolonged-shutdown/
