bug: Unable to create Lambda from S3 archive

[wszychta]

Is there an existing issue for this?

• [X] I have searched the existing issues

Current Behavior

Hello,

I'm experiencing issues with localstack above version 0.13.3. We are using Boto3 python package to do few things:

1. Creating clients for S3 and lambda

lambda_client: LambdaClient = client(service_name='lambda', endpoint_url=args.endpoint)
packaging_bucket: Bucket = resource(service_name='s3', endpoint_url=args.endpoint).Bucket(name='packaging-bucket')

2. Creating S3 bucket

bucket.create()

3. Uploading lambda ZIP file to the bucket

bucket.upload_file(
        Filename=str(package.path),
        Key=package.zip_name,
    )

4. Creating Lambda from ZIP file inside created S3 bucket

params = dict(
        FunctionName=package.function_name,
        Runtime=python_version,
        Handler=f'{package.name}.main.{handler_name}',
        Code={
            'S3Bucket': packaging_bucket.name,
            'S3Key': package.zip_name,
        },
        Role='arn:aws:iam::123456:role/irrelevant',
        Timeout=lambda_timeout,
    )
resp = lambda_client.create_function(**params)

I would say that this is a simple use case. It was working fine before upgrading localstack above 0.13.3

What we see is below error:

Traceback (most recent call last):
  File ".cicd_scripts/other/localstack_add_lambda.py", line 170, in <module>
    main()
  File ".cicd_scripts/other/localstack_add_lambda.py", line 159, in main
    create_lambda(
  File ".cicd_scripts/other/localstack_add_lambda.py", line 127, in create_lambda
    resp = lambda_client.create_function(**params)
  File "/pipeline/dev-venv/lib/python3.8/site-packages/botocore/client.py", line 395, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/pipeline/dev-venv/lib/python3.8/site-packages/botocore/client.py", line 725, in _make_api_call
    raise error_class(parsed_response, operation_name)
botocore.exceptions.ClientError: An error occurred (InternalFailure) when calling the CreateFunction operation: Unable to fetch Lambda archive from S3: An error occurred (403) when calling the HeadObject operation: Forbidden

Expected Behavior

This code should create Lambda function and give us in response something like this:

{'ResponseMetadata': {'HTTPStatusCode': 200, 'HTTPHeaders': {'content-type': 'application/json', 'content-length': '721', 'access-control-allow-origin': '*', 'access-control-allow-methods': 'HEAD,GET,PUT,POST,DELETE,OPTIONS,PATCH', 'access-control-allow-headers': 'authorization,cache-control,content-length,content-md5,content-type,etag,location,x-amz-acl,x-amz-content-sha256,x-amz-date,x-amz-request-id,x-amz-security-token,x-amz-tagging,x-amz-target,x-amz-user-agent,x-amz-version-id,x-amzn-requestid,x-localstack-target,amz-sdk-invocation-id,amz-sdk-request', 'access-control-expose-headers': 'etag,x-amz-version-id', 'connection': 'close', 'date': 'Thu, 24 Mar 2022 09:45:16 GMT', 'server': 'hypercorn-h11'}, 'RetryAttempts': 0}, 'FunctionName': 'lambda-example1', 'FunctionArn': 'arn:aws:lambda:eu-west-1:000000000000:function:lambda-example1', 'Runtime': 'python3.8', 'Role': 'arn:aws:iam::123456:role/irrelevant', 'Handler': 'lambda_example1.main.lambda_handler', 'CodeSize': 17650948, 'Description': '', 'Timeout': 30, 'LastModified': '2022-03-24T09:45:14.963+0000', 'CodeSha256': 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX', 'Version': '$LATEST', 'VpcConfig': {}, 'Environment': {'Variables': {'var1': 'value1', 'var2': 'value2'}}, 'TracingConfig': {'Mode': 'PassThrough'}, 'RevisionId': 'ef12c9f1-7af4-4102-938a-fd03fc745144', 'State': 'Active', 'LastUpdateStatus': 'Successful', 'PackageType': 'Zip', 'Architectures': ['x86_64']}

How are you starting LocalStack?

Custom (please describe below)

Steps To Reproduce

How are you starting localstack (e.g., bin/localstack command, arguments, or docker-compose.yml)

We are running localstack as a service in gitlab runners. Task container is connected with localstack container via link, so there is no connectivity issue.

It looks just like this:

task_name:
  variables:
    # Variables for Localstack
    LOCALSTACK_DEFAULT_REGION: eu-west-1
    LOCALSTACK_HOSTNAME: 127.0.0.1 # This needs to be provided since localstack v0.12.9 https://github.com/localstack/localstack/issues/3816#issuecomment-826167132
    LOCALSTACK_DEBUG: 1
    ENABLE_CONFIG_UPDATES: 1
    LOCALSTACK_URL: http://lambda-example1-localstack-ct:4566
    LOCALSTACK_SERVICES: lambda,s3,sns,sqs
    HOSTNAME_EXTERNAL: lambda-example1-localstack-ct
    LOCALSTACK_LAMBDA_ENV_VARS: '{"var1":"value1","var2":"value2"}'
  services:
    - name: localstack/localstack:0.14.1
      alias: lambda-example1-localstack-ct
  before_script:
  - | 
       python3 .cicd_scripts/other/localstack_add_lambda.py \
        --localstack-url ${LOCALSTACK_URL}                  \
        --py-version ${PY_VERSION}                          \
        --service-name ${LAMBDA_NAME}                    \
        --job-id ${CI_JOB_ID}                               \
        --env-vars ${LOCALSTACK_LAMBDA_ENV_VARS}

Client commands (e.g., AWS SDK code snippet, or sequence of "awslocal" commands)

look in Current Behavior section

Environment

- OS: 
- LocalStack: above 0.13.3 (0.14.0 and 0.14.1 so far)

Anything else?

I have tried suggestion described here, but I didn't solve my issue.

[github-actions[bot]]

Welcome to LocalStack! Thanks for reporting your first issue and our team will be working towards fixing the issue for you or reach out for more background information. We recommend joining our Slack Community for real-time help and drop a message to LocalStack Pro Support if you are a Pro user! If you are willing to contribute towards fixing this issue, please have a look at our contributing guidelines and our developer guide.

[whummer]

Thanks for reporting @wszychta . Interesting, not sure why you're getting a 403 there - are you creating the bucket with any ACLs defined, by any chance?

Wondering if the S3 HeadObject request is being routed to real AWS instead of LocalStack, which should technically not be the case (looking at our CreateFunction Lambda code..)

Can you try adding a step to your pipeline that tries to exec into the LocalStack container and attempts to download the S3 file directly via awslocal - something similar to (assuming lambda-example1-localstack-ct is the container name):

docker exec -it lambda-example1-localstack-ct awslocal s3 cp s3://packaging-bucket/path/to/handler.zip /tmp/handler.zip

Hope that helps - we'll also try debugging more with a Gitlab runner instance, to see if potentially one of your configuration options creates a conflict. Can you also try commenting out this line - should technically no longer be required with the latest version:

# LOCALSTACK_HOSTNAME: 127.0.0.1

Thanks

[wszychta]

Hello @whummer,

1. As you can see in our example, we are not adding any ACL settings to the bucket. I was trying to Allow Read All, but I also had the same issue.
2. I was able to list all files from the created bucket with S3 boto3 client. I believe that I would be also able to get file from the same bucket. Also I'm sure that it is not a real bucket, because we have only one file inside it.
3. Thanks for your suggestion. I will verify if there is no need for this variable LOCALSTACK_HOSTNAME in our environment anymore.

[whummer]

Hi again @wszychta , have you been able to give it another try in the meantime? Just to make sure, can you please confirm if the problem still persists with the latest Docker image? Thanks!

[wszychta]

Hello again @whummer. We have just confirmed that issue still exists with the latest docker image.

[dfangl]

Hi, just to make sure, could you try using the docker-compose file we provide in this repo (https://github.com/localstack/localstack/blob/master/docker-compose.yml), pull from latest, and try executing the task from outside of docker with the endpoints set to localhost? You have a lot of deprecated or dangerous variables defined, and I would want to make sure none of them is interfering with this.

[wszychta]

Hello,

It turned out that this was issue with our proxy configuration. Localstack in our case was requiring access to below urls:

• s3.localhost.localstack.cloud
• analytics.localstack.cloud
• github.com

When one of my teammates set environment to no_proxy it started to work. Can you tell me if you can think about other endpoints which needs to be accessed while using localstack? We would like to be prepared for future issues like that.

[dfangl]

We get some dependencies directly from the project source, like elasticsearch, opensearch etc. For basic operation, the above three should be enough, but for other services, it might fail.

I will close this issue as it is not related to lambda anymore, please do a request in https://github.com/localstack/docs/issues for a documentation about the exact external endpoints required! Thank you for reporting back!