Clarify import of pg database

[cep21]

Hi,

I tried deploying these manually myself and was unable to import "_pg" inside my lambda environment. It appears related to the issue at https://github.com/jkehler/awslambda-psycopg2 where

This is a custom compiled psycopg2 C library for Python. Due to AWS Lambda missing the required PostgreSQL libraries in the AMI image, we needed to compile psycopg2 with the PostgreSQL libpq.so library statically linked libpq library instead of the default dynamic link.

Can the github repository clarify if this is correct and how to build and deploy this code?

[willtong1234]

These sample Lambdas are published to the Serverless Application Repository (https://serverlessrepo.aws.amazon.com/applications), which is our preferred method of deploying these Lambda functions. It deploys with all the necessary dependencies as well. The easiest way to deploy these is to use the AWS Secrets Manager console or the CLI (directions here: https://docs.aws.amazon.com/secretsmanager/latest/userguide/enable-rotation-rds.html).

That said, I will keep this Issue open until we update the README file with more information.

[cep21]

Thank you. I had to fork the lambdas to add a complete postgres connection string to the JSON postgres://bob:password@host/database. It may be helpful, especially since these are examples, for people to know how to build and deploy it themselves.

If anyone else finds this, here is how I am packaging this lambda

# Used to build the lambda artifact
FROM postgres:10.12 as pg
FROM python:3.8.2-buster as py
RUN apt-get update && \
        apt-get install --no-install-recommends -y \
                ca-certificates=20* \
                zip=3* \
                unzip=6.* && \
        rm -rf /var/lib/apt/lists/*
WORKDIR /package
# All of these appear required by libpq
COPY --from=pg /usr/lib/x86_64-linux-gnu/libcrypto.so.1.1 lib/
COPY --from=pg /usr/lib/x86_64-linux-gnu/libpq.so.5 lib/
COPY --from=pg /usr/lib/x86_64-linux-gnu/libssl.so.1.1 lib/
COPY --from=pg /usr/lib/x86_64-linux-gnu/liblber-2.4.so.2 lib/
COPY --from=pg /usr/lib/x86_64-linux-gnu/libldap-2.4.so.2 lib/
COPY --from=pg /usr/lib/x86_64-linux-gnu/libldap_r-2.4.so.2 lib/
COPY --from=pg /usr/lib/x86_64-linux-gnu/libsasl2.so.2 lib/
COPY --from=pg /usr/lib/x86_64-linux-gnu/libgnutls.so.30 lib/
COPY --from=pg /lib/x86_64-linux-gnu/libidn.so.11 lib/
COPY --from=pg /usr/lib/x86_64-linux-gnu/libnettle.so.6 lib/
COPY --from=pg /usr/lib/x86_64-linux-gnu/libhogweed.so.4 lib/

WORKDIR /pycode
RUN python -m venv venv
COPY requirements.txt .
RUN . venv/bin/activate && pip install -r requirements.txt && deactivate

WORKDIR /package

RUN cp -r /pycode/venv/lib/python3.8/site-packages/* .
RUN ls -la /package
COPY lambda_function.py .
RUN zip -r /lambda.zip .
ENTRYPOINT ["cat", "/lambda.zip"]

Here are my requirements.txt

boto3==1.12.39
botocore==1.15.39
docutils==0.15.2
jmespath==0.9.5
psycopg2-binary==2.8.5
PyGreSQL==5.1.1
python-dateutil==2.8.1
s3transfer==0.3.3
six==1.14.0
urllib3==1.25.8

I set this up via virtual env.

Executing this docker will create the lambda zip needed for deployment.

[artburkart]

These sample Lambdas are published to the Serverless Application Repository (https://serverlessrepo.aws.amazon.com/applications), which is our preferred method of deploying these Lambda functions. It deploys with all the necessary dependencies as well. The easiest way to deploy these is to use the AWS Secrets Manager console or the CLI (directions here: https://docs.aws.amazon.com/secretsmanager/latest/userguide/enable-rotation-rds.html).

That said, I will keep this Issue open until we update the README file with more information.

Is this still true? I'm pretty confused how this is supposed to be deployed today. I found no app in the serverless repository nor was Secrets Manager able to deploy this code for me. I started deploying manually, because I saw no other way to deploy it.

[mlehner616]

Yes it's still there, when searching SAR, make sure you click the "Show apps that create custom IAM roles or resource policies" checkbox.

Edit: now that terraform can deploy SAR repos, I'd recommend doing it that way.

[srisudarsan]

Hello @willtong1234 , have there been any updates on this ?. Can we please know how this was built, as we would need this information for us to build the package. We would need it to be deployed through terraform and we are not in a position to use the SAR application

[srisudarsan]

Thank you. I had to fork the lambdas to add a complete postgres connection string to the JSON postgres://bob:password@host/database. It may be helpful, especially since these are examples, for people to know how to build and deploy it themselves.

If anyone else finds this, here is how I am packaging this lambda

# Used to build the lambda artifact
FROM postgres:10.12 as pg
FROM python:3.8.2-buster as py
RUN apt-get update && \
        apt-get install --no-install-recommends -y \
                ca-certificates=20* \
                zip=3* \
                unzip=6.* && \
        rm -rf /var/lib/apt/lists/*
WORKDIR /package
# All of these appear required by libpq
COPY --from=pg /usr/lib/x86_64-linux-gnu/libcrypto.so.1.1 lib/
COPY --from=pg /usr/lib/x86_64-linux-gnu/libpq.so.5 lib/
COPY --from=pg /usr/lib/x86_64-linux-gnu/libssl.so.1.1 lib/
COPY --from=pg /usr/lib/x86_64-linux-gnu/liblber-2.4.so.2 lib/
COPY --from=pg /usr/lib/x86_64-linux-gnu/libldap-2.4.so.2 lib/
COPY --from=pg /usr/lib/x86_64-linux-gnu/libldap_r-2.4.so.2 lib/
COPY --from=pg /usr/lib/x86_64-linux-gnu/libsasl2.so.2 lib/
COPY --from=pg /usr/lib/x86_64-linux-gnu/libgnutls.so.30 lib/
COPY --from=pg /lib/x86_64-linux-gnu/libidn.so.11 lib/
COPY --from=pg /usr/lib/x86_64-linux-gnu/libnettle.so.6 lib/
COPY --from=pg /usr/lib/x86_64-linux-gnu/libhogweed.so.4 lib/

WORKDIR /pycode
RUN python -m venv venv
COPY requirements.txt .
RUN . venv/bin/activate && pip install -r requirements.txt && deactivate

WORKDIR /package

RUN cp -r /pycode/venv/lib/python3.8/site-packages/* .
RUN ls -la /package
COPY lambda_function.py .
RUN zip -r /lambda.zip .
ENTRYPOINT ["cat", "/lambda.zip"]

Here are my requirements.txt

boto3==1.12.39
botocore==1.15.39
docutils==0.15.2
jmespath==0.9.5
psycopg2-binary==2.8.5
PyGreSQL==5.1.1
python-dateutil==2.8.1
s3transfer==0.3.3
six==1.14.0
urllib3==1.25.8

I set this up via virtual env.

Executing this docker will create the lambda zip needed for deployment.

While trying this, I am getting an error "libidn.so.11 is not found", Is there something I am missing here ?

[goyalya]

@srisudarsan: SAR templates can be deployed via Terraform. You can also pull the dependencies from the zipfile as described by @willtong1234 above.