Using lambda layers to provide missing dependencies

[Leooo]

Hello, I don't think this one has been raised before:

I need to use the "Klayers-python37-requests" layer to be able to use the requests library in my lambdas.

I can fetch the layer in moto (actually download and then zip the downloaded version), and create lambdas with this layer, but if I'm not wrong it's not actually providing the actual dependency when running the lambdas? In practice, I see No module named 'requests' everywhere when running the lambdas.

Request is: can we run the code provided by the layers to make the libraries available to the running code in the lambdas

[bblommers]

Hi @Leooo, thanks for raising this. You're right, we're not actually doing anything with layers atm.

Not sure how we would implement this, as it requires AWS credentials to call get-layer-version-by-arn for each layer. So we can't get the download-location within Moto, as it's supposed to always run without providing creds.

Will mark it as an enhancement though, in case anyone has any ideas.

[Leooo]

@bblommers I am able to download the layers currently, i.e. physically download the zip file and put them in my test folder. And then create lambdas with such layers. So that's not really the problem (or at least the biggest priority). Main point to me is that adding this layer to a lambda basically doesn't expose the corresponding libraries to the code so that the lambda code can use import xxx where xxx is defined in the layer.

[Leooo]

So we just can't run any lambdas in moto currently if they use a layer (except if we just run the function straight without any wrapper like step functions or cloudformation). That seems a big blocker to me.

[bblommers]

Yeah, I think we're talking about the same thing, I just didn't explain it very well.

The easiest way to fix your problem:

• user passes in the layer when creating the Lambda (as usual)
• Moto finds the location of the layer
• Moto downloads the layer at runtime/testtime
• unzips it
• passes the dependencies to the Docker-container running the Lambda-code

The problem with the second step, if we use the get-layer-version-by-arn-method to find the location of the layer, is that it requires credentials.

So the alternatives could be:

• Let the user download the layers manually (like you did) and pass the location on the FS to moto
• Pass the S3-url to moto where the layer is stored

(I only had limited exposure to layers, but I think I'm correct when assuming that all layers are stored in S3?)

[Leooo]

Yes for both cases @bblommers , but again even if we do that, atm we have the problem that the layer is not really consumed in by the lambda at all? (or I'm missing something).

Here is my code to make things more clear:

        # URL=$(aws lambda get-layer-version-by-arn --arn arn:aws:lambda:eu-west-1:113088814899:layer:Klayers-python37-requests:14 --query Content.Location --output text --profile=adfs)
        layer_name='Klayers-python37-requests'
        self.client_lambda.publish_layer_version(
            LayerName=layer_name,
            Content={"ZipFile": open('request_layer.zip', 'rb').read()}, # layer saved locally
            CompatibleRuntimes=["python3.7"],
            LicenseInfo="MIT",
        )
        request_layer_arn = f'arn:aws:lambda:{LAMBDA_REGION}:{AWS_ACCOUNT_NUMBER}:layer:{layer_name}:1'
            return self.client_lambda.create_function(
                FunctionName=lambda_name,
                Runtime="python3.7",
                Role=self.client_cf.default_role,
                Handler=f'python_lambdas.{lambda_name}.lambda_handler',
                #Code={"ZipFile": lambda_zip_files[lambda_name]},
                Code={'ZipFile': open('lambdas_zipped.zip', 'rb').read()}, #https://stackoverflow.com/a/47599666/1984997
                Description=f'{lambda_name} test lambda function',
                Timeout=3,
                MemorySize=128,
                Publish=True,
                Layers=[
                    request_layer_arn
                ]
            )['FunctionArn']

=> in the above, the lambda is created with the layer, no huge problem here, but then it looks like the layer is not actually accessed in the lambda code (No module named 'requests' error everywhere when running the lambda.)

[bblommers]

Correct, the next steps still need to be implemented as well in order for this to work:

• Moto downloads the layer at runtime/testtime - unzips it - passes the dependencies to the Docker-container running the Lambda-code

[neel-gala]

@Leooo @bblommers Do we currently have any workaround for this problem?

[shubh1205]

Has this been fixed yet? I am still struggling with this issue and it seems like huge miss to me because I can't really use any external package while testing.

[bblommers]

@neel-gala @shubh1205 As far as I know, no one has started working on this yet

[LaineWish]

Hello! Just wondering if there are any updates on this?

[karkir0003]

@Leooo, how were you able to resolve this issue. I have some code where I'm trying to use a pandas layer for python 3.9 in moto. I use pd.__file__ to get the file, upload to a mocked s3 bucket, and read from it (by passing S3 Bucket, S3 key in the publish_layer_version function). When I pass that ARN into the lambda mock (create_function), I get an error with no module named pandas

[Leooo]

@karkir0003 it's too old an issue for me to remember the details. Looking at the thread it looks like there doesn't seem to be any easy workaround.