Have to reset ALB target group lambda:InvokeFunction permissions on each update

[gigi-at-zymergen]

Context

We use Zappa in our Jenkins CI pipelines to deploy/update Lambdas. We're using target groups in ALBs to invoke Lambdas in a hybrid environment where some rules forward requests to Lambdas, and other requests go to EC2 or k8s hosted services. It's entirely internal, and we use ALBs rather than the API Gateway to declaratively manage our hybrid routing rules. We have had to add a step to all our pipelines to reset permissions to allow InvokeFunction from ALB target groups after every update:

aws lambda add-permission --function-name ${lambda-ARN} \
--action lambda:InvokeFunction --statement-id elb1 --principal elasticloadbalancing.amazonaws.com

Is there some configuration we could use to preserve this permission from update to update? Seems related to https://github.com/Miserlou/Zappa/issues/980 , but triggering from ALB Target Groups is somewhat different than the API Gateway.

We're also not using Zappa to configure an ALB ( using an existing ALB configured externally ) so this may also be related to https://github.com/Miserlou/Zappa/issues/1832 . And permissions may get overwritten with the assumption that Zappa would be handling any ALB config.

Expected Behavior

We were expecting the InvokeFunction permissions to be preserved on update.

Actual Behavior

Any existing permissions are overwritten on update

Possible Fix

We may be unaware of an existing configuration which would preserve permissions on update, but if it doesn't exist a flag to preserve existing permissions on update would be desired.

Steps to Reproduce

1. Deploy a simple zappa lambda (hello world)
2. Call aws lambda add-permission --function-name ${lambda-ARN} --action lambda:InvokeFunction --statement-id elb1 --principal elasticloadbalancing.amazonaws.com
3. configure a simple ALB with one target group to trigger the lambda
4. Test that you can trigger the lambda by calling the endpoint through the ALB
5. Use zappa update $VERSION_CODE -s settings.json to update the Lambda
6. Call the endpoint through the ALB and get a 502, and lambda does not trigger
7. Call the AWS add-permission call from step 2 again
8. call the endpoint and get a 200 and lambda triggers

Your Environment

• Zappa version used: 0.48.2
• Operating System and Python version: Python 3.6.4, Ubuntu 16.04
• The output of pip freeze:

argcomplete==1.9.3
boto3==1.5.8
botocore==1.8.22
certifi==2019.3.9
cfn-flip==1.2.0
chardet==3.0.4
Click==7.0
cx-Oracle==7.1.3
docopt==0.6.2
docutils==0.14
durationpy==0.5
ecdsa==0.13.2
elasticsearch==7.0.1
et-xmlfile==1.0.1
Flask==1.0.2
Flask-Cors==3.0.6
future==0.16.0
hjson==3.0.1
idna==2.7
itsdangerous==1.1.0
jdcal==1.3
Jinja2==2.10.1
jmespath==0.9.3
kappa==0.6.0
lambda-packages==0.20.0
MarkupSafe==1.1.1
mysql-connector-python==8.0.16
odfpy==1.3.6
openpyxl==2.4.9
placebo==0.9.0
protobuf==3.7.1
pyasn1==0.4.5
PyMySQL==0.8.0
python-dateutil==2.6.1
python-jose==3.0.1
python-slugify==1.2.4
PyYAML==3.13
requests==2.20.0
rsa==4.0
s3transfer==0.1.12
six==1.11.0
tablib==0.12.1
toml==0.10.0
tqdm==4.19.1
troposphere==2.4.6
unicodecsv==0.14.1
Unidecode==1.0.23
urllib3==1.24.3
Werkzeug==0.14
wsgi-request-logger==0.4.6
xlrd==1.1.0
xlwt==1.3.0
zappa==0.48.2

• Your zappa_settings.py:

{
    "v0": {
        "project_name": "name",
        "lambda_description": "desc",
        "app_function": "APP",
        "memory_size": 3008,
        "slim_handler": false,
        "role_name": "role",
        "role_arn": "REDACTED",
        "keep_warm_expression": "rate(45 minutes)",
        "keep_warm": true,
        "timeout_seconds": 30,
        "touch_path": "/",
        "aws_region": "us-west-2",
        "profile_name": "default",
        "runtime": "python3.6",
        "s3_bucket": "REDACTED",
        "manage_roles": false,
        "delete_local_zip": false,
        "delete_s3_zip": false,
        "exclude": [
            "jenkins_zappa_settings.json",
            "zappa_settings.json",
            "boto3*",
            "botocore*"
        ],
        "cloudwatch_log_level": "INFO",
        "cloudwatch_data_trace": true,
        "vpc_config": {
            "SubnetIds": [
                REDACTED
            ],
            "SecurityGroupIds": [
                "REDACTED"
            ]
        },
        "layers": [
            "arn:aws:lambda:us-west-2:703076520410:layer:oracle-libs-layer:1"
        ],
        "aws_environment_variables": {
            REDACTED
        }
    }
}

[xealot]

This is frustrating, I don't see a way to manage the ALB outside of Zappa and have the Function Policy be updated appropriately.

[adam-harwood]

I got halfway through opening a PR to fix this, but I think we're just going to migrate off Zappa instead. Still, dropping the change I hacked in that fixed this in case someone wants to pick it up. I changed deploy_lambda_alb in core.py to have:

        kwargs = dict(
            Action="lambda:InvokeFunction",
            FunctionName=lambda_name,
            Principal="elasticloadbalancing.amazonaws.com",
            SourceArn=target_group_arn,
            StatementId=lambda_name,
        )
        response = self.lambda_client.add_permission(**kwargs)

        # Register target group to lambda association.
        # https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/elbv2.html#ElasticLoadBalancingv2.Client.register_targets
        kwargs = dict(
            TargetGroupArn=target_group_arn,
            Targets=[{"Id": lambda_arn}],
        )
        response = self.elbv2_client.register_targets(**kwargs)

The key parts being the FunctionName on the add_permission call, and then the Id on the register_targets call. For reasons I don't fully understand, the code already there wasn't adding any permission (confirmed by adding a call to get the policy in there), even though it was returning a 201. With the above change it works. The part I hadn't figured out was how to include Alias in there.

Hope this helps someone.