aws-cdk-lib/aws-ec2: Egress rule to any IP not created

[chris-adam-b12]

Describe the bug

When I assign a target group to an existing application load balancer listener, the outbound rules of the ELBv2 security group are changed to allow outbound traffic only to my ECS service. However, if I ask to add an egress rule to the security group of the ELBv2 to any IPv4 or IPv6, no rule is created. Nonetheless, the sg rules appear in the cdk diff.

Adding an Egress rule to an other security group or a specific IP address does work as expected.

Expected Behavior

The security group outbound rules are created and the current egress rules are not erased.

Current Behavior

All Egress rules are erased and only the Egress rule to the ECS security group is left.

Reproduction Steps

Edit every place where there is a TODO

import * as cdk from 'aws-cdk-lib';
import * as ec2 from "aws-cdk-lib/aws-ec2";
import * as ecs from "aws-cdk-lib/aws-ecs";
import * as elbv2 from "aws-cdk-lib/aws-elasticloadbalancingv2"
import { Duration } from 'aws-cdk-lib';
import { RetentionDays } from 'aws-cdk-lib/aws-logs';

export class IacStack extends cdk.Stack {
  constructor(scope: cdk.App, id: string, props?: cdk.StackProps) {
    super(scope, id, props);
    
    const vpc = ec2.Vpc.fromLookup(this, "VPC", {
      region: "eu-central-1",
      vpcId: "vpc-123456", // TODO add your VPC ID
    });
    
    const cluster = new ecs.Cluster(this, 'cluster', { vpc: vpc });
    
    const taskDefinition = new ecs.TaskDefinition(this, 'nginxTaskDefinition', {
      compatibility: ecs.Compatibility.FARGATE,
      cpu: "256",
      memoryMiB: "512",
      networkMode: ecs.NetworkMode.AWS_VPC,
    });
    
    const nginxContainer = taskDefinition.addContainer("nginxContainer", {
      logging: ecs.AwsLogDriver.awsLogs({ streamPrefix: 'nginx', mode: ecs.AwsLogDriverMode.NON_BLOCKING, logRetention: RetentionDays.ONE_YEAR }),
      image: ecs.ContainerImage.fromRegistry("nginx"),
      portMappings: [{ containerPort: 80 }],
    });
    
    const serviceSg = new ec2.SecurityGroup(this, "ecsServiceSg", { vpc: vpc });
    const service = new ecs.FargateService(this, 'ecsService', {
      cluster,
      taskDefinition,
      desiredCount: 1,
      circuitBreaker: { rollback: true },
      enableExecuteCommand: true,
      serviceName: "nginx",
      assignPublicIp: true,
      securityGroups: [serviceSg],
    });
    
    // TODO add your ELBv2 listener ARN
    const listener = elbv2.ApplicationListener.fromLookup(this, "listener", {
      listenerArn: "arn:aws:elasticloadbalancing:eu-central-1:1234456789:listener/qwertyuiop",
    });
    const targetGroup = new elbv2.ApplicationTargetGroup(this, 'nginxTargetGroup', {
      port: 80,
      targets: [service],
      deregistrationDelay: Duration.seconds(10),
      vpc,
    });
    targetGroup.configureHealthCheck({
      enabled: true,
      path: "/",
    });
    listener.addTargetGroups("listenerRule", {
      targetGroups: [targetGroup],
      priority: 1000,
    });
    
    // TODO add your ELBv2 security group ID
    const lbSg = ec2.SecurityGroup.fromLookupById(this, "lbSg", "sg-qwerty12345");
    lbSg.connections.allowToAnyIpv4(ec2.Port.allTraffic());
    lbSg.addEgressRule(
      ec2.Peer.anyIpv6(),
      ec2.Port.allTraffic(),
      "Allow access to every ECS clusters"
    );
  }
}

Possible Solution

No response

Additional Information/Context

No response

CDK CLI Version

2.41.0 (build 6ad48a3)

Framework Version

No response

Node.js Version

v14.19.1

OS

MacOS 12.0.1

Language

Typescript

Language Version

TypeScript 3.9.10

Other information

No response

[peterwoodworth]

This may be happening due to the order CloudFormation is creating resources. The rules are showing in cdk diff you say, so you could add a dependsOn clause to ensure they're deployed after it's been initially modified. Try this?

lbSg.node.addDependency(targetGroup)

[chris-adam-b12]

Hello @peterwoodworth I tried your solution. However, the behaviour didn't change at all. I thinks the dependency should be more on the addTargetGroups action than the target group itself. I'm not even sure it's possible and it would solve the problem.

[peterwoodworth]

So when you are deploying this - in the console can you see what order resources are created / updated in? Can you verify if the egress rules are getting created before adding the target groups to the listener?

[chris-adam-b12]

Hello @peterwoodworth ,

The egress rules are indeed created before the creation of the ECS service, thus adding the target group to the load balancer. The behaviour is the same with the dependency you suggested above.

I tried to add a dependency on the service instead of the target group and it does work. Thank you for your investigation which helped me solve this issue.

I still consider this is a bug since I import a security group with fromLookup and adding a target group removes my egress rules (which I want to keep). What strange is that adding an egress rule to any IP requires a dependency rule but not an egress rule to a specific IP address or security group.

[peterwoodworth]

I'm not certain there's going to be anything we can do about ELB removing the egress rules. I think the service is doing that, but I'm not 100% positive since I can't find anything regarding this in the documentation. And there aren't any configuration settings I can find in CloudFormation either. Maybe someone else might know more about this or if we can work around this?

[corymhall]

It looks like this how the AWS::EC2::SecurityGroupEgress (or Ingress) cloudformation resource works. Before adding a new rule it will first remove any allow all rules. This is because security groups are created with an allow all rule by default and it doesn't distinguish between the default rule and user created rules.

From the CDK side of things there are a couple of issues. When using fromLookupById it will lookup the security group and set mutable: false (not allow adding additional rules) only if there is an allow all rule for both ipv4 and ipv6. But currently it is not possible to create a security group with allow all to ipv6 (linked issue).

It looks like there are two workarounds:

1. Use the escape hatch mentioned in the linked issue to add ipv6 allow all so that fromLookupById will set mutable: false.
2. Use fromSecurityGroupId instead and explicitely set mutable: false.

const lbSg = ec2.SecurityGroup.fromSecurityGroupId(this, "lbSg", "sg-0f728f9887be3db93", {
	mutable: false,
});

[github-actions[bot]]

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see. If you need more assistance, please either tag a team member or open a new issue that references this one. If you wish to keep having a conversation with other community members under this issue feel free to do so.