[ECS] [request]: Update (add/remove) ALB TargetGroups on a ECS Service that has been already created

[gfysaris]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Tell us about your request After an ECS Service is created there is no option to update/modify the ALB TargetGroups that are integrated with the service. In case there is a need to change the alb, running ECS Services need also to be replaced.

Which service(s) is this request for? ECS (Both over ASG or Fargate)

Tell us about the problem you're trying to solve. What are you trying to do, and why is it hard? We would like to have the ability to change the ALB and the TargetGroups that are integrated with an ECS Service that is already created previously.

Are you currently working around this issue? Currently, when we introduce a new ALB (and new TargetGroups for that ALB) we provision new ECS Services from scratch.

Additional context Anything else we should know? - No, nothing else.

Attachments -NONE-

[zbintliff]

One use case here is for custom deployment controllers. By allowing a service update we can attach it to a non production load bearing TG, run smoke tests, then move it over to the production TG attached to the ALB/NLB

[zivyatziv]

More information I can add - While updating a service's network configuration and adding a TG, you get an error: Load balancing settings can only be set on service creation.

Documentation is also misleading:

• https://docs.aws.amazon.com/cli/latest/reference/ecs/update-service.html
• https://docs.aws.amazon.com/AmazonECS/latest/developerguide/register-multiple-targetgroups.html

But it is described in ECS "Updating a service": https://docs.aws.amazon.com/AmazonECS/latest/developerguide/update-service.html ^ Under the important section, I quote:

If your service uses a load balancer, the load balancer configuration defined for your service when it was created cannot be changed. If you update the task definition for the service, the container name and container port that were specified when the service was created must remain in the task definition.

To change the load balancer name, the container name, or the container port associated with a service load balancer configuration, you must create a new service.

Amazon ECS does not automatically update the security groups associated with Elastic Load Balancing load balancers or Amazon ECS container instances.

Question is about the 3rd point I've quoted (4th in the page), why? :(

[andrewloux]

This is one of the biggest painpoints using ECS on Fargate

[anujkumar-df]

Can anyone try detaching the target group from one load balancer and attaching it to another load balancer? I just found that using this method, you can basically change the load balancer endpoint. You don't even need to modify the existing ECS service.

[AbhishekNautiyal]

Amazon ECS UpdateService API now supports updates for the following parameters: loadBalancers, propagateTags, enableECSManagedTags, and serviceRegistries; see what's new post for details:

https://aws.amazon.com/about-aws/whats-new/2022/03/amazon-ecs-service-api-updating-elastic-load-balancers-service-registries-tag-propagation-ecs-managed-tags/

[kevinkupski]

When updating the Load Balancer field and using a classic load balancer, I get InvalidParameterException: Target Group Arn can not be blank.. Is this something you see as well?

[AbhishekNautiyal]

When updating the Load Balancer field and using a classic load balancer, I get InvalidParameterException: Target Group Arn can not be blank.. Is this something you see as well?

Hi @kevinscholz could you share more details about your configuration. Are you trying to add a load balancer to a service that did not have one previously? It would be great if you could also DM me your service details and updateService payload. Thanks.

[kevinkupski]

It appeared when updating the Terraform AWS Provider. One of the tests failed, which basically first creates a classic load balancer and attaches it to a service and then updates the container name and container port in the load balancer attachment.

The interesting part is this:

resource "aws_ecs_service" "test" {
  name            = %[1]q # This is fix and does not change
  cluster         = aws_ecs_cluster.test.id
  task_definition = aws_ecs_task_definition.test.arn
  desired_count   = 1
  iam_role        = aws_iam_role.ecs_service.name
  load_balancer {
    elb_name       = aws_elb.test.id
    container_name = %[3]q # this changes during the update from "ghost" to "nginx"
    container_port = %[4]d # this changes during the update from "2368" to "80"
  }
  ...
}

Do you need additional information? Not sure how to reach you via DM.

[AbhishekNautiyal]

It appeared when updating the Terraform AWS Provider. One of the tests failed, which basically first creates a classic load balancer and attaches it to a service and then updates the container name and container port in the load balancer attachment.

The interesting part is this:

resource "aws_ecs_service" "test" {
  name            = %[1]q # This is fix and does not change
  cluster         = aws_ecs_cluster.test.id
  task_definition = aws_ecs_task_definition.test.arn
  desired_count   = 1
  iam_role        = aws_iam_role.ecs_service.name
  load_balancer {
    elb_name       = aws_elb.test.id
    container_name = %[3]q # this changes during the update from "ghost" to "nginx"
    container_port = %[4]d # this changes during the update from "2368" to "80"
  }
  ...
}

Do you need additional information? Not sure how to reach you via DM.

Hi @kevinscholz We're looking into it. We'd need some more details to investigate (viz. account number, region, timeframe of failure). Could you please send this info over an email at nautiya[at]amazon[.]com or open a ticket with AWS Support and share the ticket ID with me?

[genbit]

The issue is happening when updating a service using CloudFormation (CFN). The reference to TargetGroup is not removed. I'm reopening the issue to track the related CFN improvements in one place.

[bvtujo]

For customers using the CDK to manage ECS services, you can override the default behavior which sets an empty LoadBalancers field to undefined.

From a cloudformation template:

declare const yourServiceResourceName string;
const template = new cdk.cloudformation_include.CfnInclude(scope, 'Template', {
  templateFile: 'path/to/template.yml',
});
const svc = this.template.getResource(yourServiceResourceName) as ecs.CfnService;
svc.loadBalancers = [];

Or from an existing CDK construct:

declare const stack: cdk.IStack;
declare const cluster: ecs.ICluster;
declare const taskDefinition: ecs.IFargateTaskDefinition;
const svc = new ecs.FargateService(stack, 'FargateService', {
  cluster,
  taskDefinition,
});

(svc.node.defaultChild as ecs.CfnService).loadBalancers = [];

[monelgordillo]

This becomes an issue with ECS, load balancers, and Code Deployment Groups.

[monelgordillo]

Any update on this? 2020 since this issue was open.

[karanbokil]

The issue is happening when updating a service using CloudFormation (CFN). The reference to TargetGroup is not removed. I'm reopening the issue to track the related CFN improvements in one place.

Hi @genbit, are you experiencing this issue with Application Load Balancer or Network Load Balancer or Gateway Load Balancer? I have verified that changing/adding/removing TargetGroups in a CFN resource for Service works as expected with ALBs, would be useful to know what type of configuration you are running?

Any update on this? 2020 since this issue was open.

Hi @monelgordillo, we did address the original issue per this release, but it seems there are reports of some specific edge cases, could you clarify what scenario is breaking for you?

It appeared when updating the Terraform AWS Provider. One of the tests failed, which basically first creates a classic load balancer and attaches it to a service and then updates the container name and container port in the load balancer attachment.

The interesting part is this:

resource "aws_ecs_service" "test" {
  name            = %[1]q # This is fix and does not change
  cluster         = aws_ecs_cluster.test.id
  task_definition = aws_ecs_task_definition.test.arn
  desired_count   = 1
  iam_role        = aws_iam_role.ecs_service.name
  load_balancer {
    elb_name       = aws_elb.test.id
    container_name = %[3]q # this changes during the update from "ghost" to "nginx"
    container_port = %[4]d # this changes during the update from "2368" to "80"
  }
  ...
}

Do you need additional information? Not sure how to reach you via DM.

Hi @kevinkupski, is this issue still persistent? Would like to understand if this is a terraform provider fix needed to include the update capability post-release or not?

[innokenty]

I have just now experienced an issue, that I believe might be the same, or closely related. @karanbokil

Setup

I am using CodePipeline + raw CloudFormation (yamls) to deploy a simple ECS Fargate service, which runs a java microservice as a single Docker task. It has a network interface, that only implements a single health-check endpoint.

The service also had an ElasticLoadBalancingV2::LoadBalancer using AWS::ElasticLoadBalancingV2::Listener to reference a AWS::ElasticLoadBalancingV2::TargetGroup, using the service's health check endpoint. The ECS service definition was referencing the same TargetGroup under LoadBalancers property:

  Service:
    Type: AWS::ECS::Service
    Properties:
      LoadBalancers:
        - ContainerName: !Ref ContainerName
          ContainerPort: !Ref ContainerPort
          TargetGroupArn: !Ref TargetGroupArn
      LaunchType: FARGATE
      ...

Let me know if you want me to provide code snippets of the ELB also, I just think this is very standard setup.

The Issue

The issue happened when I removed the LoadBalancer entirely (it was all working fine, but actually wasn't used). I removed all of the above (ELB & Co and LoadBalancers section from the service) and what happened is that after running CloudFormation deploy – it has successfully deployed everything, deleting all the ELB resources, and the TargetGroup and pretending to update the ECS Service, but in fact it didn't remove the LoadBalancers section from the service definition!

It just simply didn't! It was gone from my CloudFormation template, but when I would open the ECS service in Console UI – I would get an error saying: One or more target groups not found, because well – it was actually deleted correctly. Redeploys of the ECS Service didn't help (they would get stuck and never finish), using the UI I couldn't solve or even see the issue.

Redeploying a new task version with CloudFormation (to force service update) also failed with:

The following resource(s) failed to update: [MyService]. (that is the AWS::ECS::Service resource)

Resource handler returned message: "Invalid request provided: UpdateService error: The target group arn:aws:elasticloadbalancing:eu-west-1:1234567890:targetgroup/my-service/12345qwerty67890 does not exist. (Service: AmazonECS; Status Code: 400; Error Code: InvalidParameterException; [...] HandlerErrorCode: InvalidRequest)

(Redeploying CF without updating the service of course went through, because it was just skipping the template with no changes)

Fix

The only way I could fix the service is to manually update it via AWS CLI – worked, thanks to the article linked somewhere above!

I first used ecs describe-service and could see the deleted-not-actually-deleted load balancer under the "loadBalancers": [....] property. Then I used the ecs update-service --load-balancers '[]' command, which removed the broken piece, the service restarted by itself and everything was fine again.

Conclusion

I think the issue is somewhere between CloudFormation and ECS integration, that it simply doesn't update the LoadBalancers section of the service somehow, while thinking it actually did (my CF was all green and deployment went through).

[karanbokil]

Thanks @innokenty for the additional context! We will be looking into this and provide updates on the fix

[djchino3]

Has there been any traction on this issue? it seems to still be affecting customers.