SNOW-358871: Does not work in AWS lambda - Dead on Arrival

[imperialguy]

Please answer these questions before submitting your issue. Thanks!

1. What version of Python are you using (python --version)? 3.7 or 3.8

2. What operating system and processor architecture are you using (python -c 'import platform; print(platform.platform())')? AWS lambda amazon linux 2

3. What are the component versions in the environment (pip list)?

Package                    Version
-------------------------- ---------
asn1crypto                 1.4.0
azure-common               1.1.27
azure-core                 1.14.0
azure-storage-blob         12.8.1
boto3                      1.17.75
botocore                   1.20.75
certifi                    2020.12.5
cffi                       1.14.5
chardet                    3.0.4
cryptography               3.4.7
greenlet                   1.1.0
idna                       2.10
isodate                    0.6.0
jmespath                   0.10.0
msrest                     0.6.21
oauthlib                   3.1.0
oscrypto                   1.2.1
pip                        21.1.1
pycparser                  2.20
pycryptodomex              3.10.1
PyJWT                      2.1.0
pyOpenSSL                  20.0.1
python-dateutil            2.8.1
pytz                       2021.1
requests                   2.25.1
requests-oauthlib          1.3.0
s3transfer                 0.4.2
setuptools                 47.1.0
six                        1.16.0
snowflake-connector-python 2.4.3
snowflake-sqlalchemy       1.2.4
SQLAlchemy                 1.4.15
urllib3                    1.26.4
wheel                      0.36.2

4. What did you do? Step 1: Install snowflake-sqlalchemy as an AWS lambda layer. Step 2: Create a sample lambda function with the following code. Add the above layer to it and run the function.

   import snowflake.sqlalchemy
   
   def lambda_handler(event, context):
      print("hello world!')
   

5. What did you expect to see? I expect to see "hello world" and the AWS lambda function executed without any issue

6. What did you see instead?

Response
{
  "errorMessage": "Unable to import module 'lambda_function': No module named 'snowflake.sqlalchemy'",
  "errorType": "Runtime.ImportModuleError",
  "stackTrace": []
}

7. Can you set logging to DEBUG and collect the logs? Not sure how the logging works on AWS lambda as per your below suggestion. I tried, but it didn't work.

import logging
import os

for logger_name in ['snowflake.sqlalchemy', 'snowflake.connector', 'botocore']: 
    logger = logging.getLogger(logger_name)
    logger.setLevel(logging.DEBUG)
    ch = logging.StreamHandler()
    ch.setLevel(logging.DEBUG)
    ch.setFormatter(logging.Formatter('%(asctime)s - %(threadName)s %(filename)s:%(lineno)d - %(funcName)s() - %(levelname)s - %(message)s'))
    logger.addHandler(ch)

8. Alternate test If you remove the line import snowflake.sqlalchemy and replace it with (say) import snowflake.connector, it works fine. So, that tells me that the problem is not with the whole layer, which comprises several libraries like sqlalchemy, snowflake-connector etc., but it is specifically with snowflake-sqlalchemy package only.

[GreekPoet]

I saw this issue because I was working on the same exact work. I just want to note that if you are building python packages for lambda, you have to install and zip up the packages on the same OS as lambda. If you do it on OSX for example, it will not have the proper header files to run in an AWS linux AMI for example.

[imperialguy]

The packages were zipped up on Amazon Linux 2 - the same that is used by lambda.

[sfc-gh-ppietrzkiewicz]

My customer is seeing the same: I use following code to zip dependencies which worked well for any other dependency like sqlalchemy, but it does not work for snowflake-sqlalchemy

cd ~/GitHub/[MYPROEJCT]/shared/AWS_Lambda_Layer
export lib=snowflake
export package=snowflake-sqlalchemy
export bucket=[MYBUCKET]
mkdir $lib
cd $lib
 
pip install $package --target ./.deployment
cd .deployment
zip -m -r $lib.zip .
 

copy zipped file to S3 aws s3 cp $lib.zip s3://$bucket/lambda-layer/ publish lambda layer aws lambda publish-layer-version --layer-name $lib --content S3Bucket=$bucket,S3Key=lambda-layer/$lib.zip --compatible-runtimes python3.9 --compatible-architectures arm64

Then I add the snowflake layer to tested lambda function and it fails with following

When I don’t use layers and package snowflake-sqlalchemy dependencies during lambda deployment, it works well. When I use layers for sqlalchemy dependencies and package snowflake connector to the lambda, it works well and correctly picks up dependencies from layers.

I tried changing the order of merging as well as names of layers – none of that has effect on either functionality

This is a snippet of the dependencies

[ettoreaquino]

Any updates on this issue?

Exact same behavior as reported by @imperialguy when creating a Lambda Layer running on AWS Lambda Python3.8 runtime

[Srinu3366]

It is working for me with lambda runtime Python3.9 if I build the layer inside a container using image public.ecr.aws/sam/build-python3.9:latest-x86_64.

[nnimis]

I can confirm this is working with Python 3.9, I wasn't able to make it work with 3.8

[sfc-gh-dszmolka]

marking this as closed as it seems to be resolved - if still further troubleshooting needed, do comment please and i can reopen