Python 3.11 support

[grantmcconnaughey]

AWS Lambda now supports Python 3.11. It would be great if this package was updated to support Python 3.11, since it has some really nice performance improvements.

https://aws.amazon.com/about-aws/whats-new/2023/07/aws-lambda-python-3-11/

[danthemanvsqz]

I was able to get the 3.9 module to run on 3.11 by renaming the file _psycopg.cpython-39-x86_64-linux-gnu.so to _psycopg.so. I then zipped it up and created a layer

[isaaclall]

could someone help exactly how do I get this to work with python3.11?

[danthemanvsqz]

@isaaclall Rename this file to _psycopg.so. Then zip the directory in the layer structure that you use e.g. /python or site-packages And deploy. You might want to use ssl to connect and in that case follow these instructions

[isaaclall]

ahh thank you!

[anuragdev101]

Can you confirm if this is working for you, for 3.11 python with openssl

[romor]

Can you confirm if this is working for you, for 3.11 python with openssl

Does not work for me. I get the following error message: [ERROR] Runtime.ImportModuleError: Unable to import module 'store_parameter': libssl.so.3: cannot open shared object file: No such file or directory

[zosiek666]

I have create a PR with changes that allow me to use psycopg2 in AWS lambdas https://github.com/jkehler/awslambda-psycopg2/pull/84

[danthemanvsqz]

@zosiek666 does this have ssl support?

[avaki11]

Any update on this solutions for ssl support?

@zosiek666 does this have ssl support?

[abrivio2]

@zosiek666 Could you explain to us how you did the build with the ssl support? How did you import the libssl.so library?

[grantmcconnaughey]

If it helps, me and my coworker got this working for my startup Postpone. Here are the notes we took:

How to Build Psycopg2 for AWS Lambda

The Psycopg2 package is unfortunately not natively supported by AWS lambda environments. For that reason, we need to compile and vendor it in our backend application code. The binary along with its Python sources can be found in the backend/psycopg2 folder.

Steps

Create a local AWS Docker container

We first need an environment as similar as possible to AWS Lambda’s where we will compile Postgres and Psycopg2. For that, follow the instructions from here, which are summarized below:

1. Create a new folder called build_target and cd into it.

$ mkdir build_target
$ cd build_target

2. Create a new lambda_function.py file:

import sys
def handler(event, context):
    return 'Hello from AWS Lambda using Python' + sys.version + '!'

3. Create an empty requirements.txt file:

$ touch requirements.txt

4. Create a new Dockerfile with this content:

FROM public.ecr.aws/lambda/python:3.11
# Copy requirements.txt
COPY requirements.txt ${LAMBDA_TASK_ROOT}
# Copy function code
COPY lambda_function.py ${LAMBDA_TASK_ROOT}
# Install the specified packages
RUN pip install -r requirements.txt
# Set the CMD to your handler
CMD [ "lambda_function.handler" ]

VOLUME [ "/files" ]

5. Build the container (this command fails randomly with "ERROR: failed to solve: public.ecr.aws/lambda/python:3.11: failed to authorize."):

$ docker build -t aws-lambda:build .

6. Download and extract the Postgres sources into the build_target folder. You can find them here.

7. Download and extract the psycopg2 sources into the build_target folder. You can find them here.

8. Run the container, mounting the current build_target folder into the container’s /files volume:

$ docker run -v $(pwd):/files --name aws_build_container

9. Start a bash session in the container:

$ docker exec -it aws_build_container bash

10. Go into the /files folder and make sure the Postgres and Psycopg2 sources are there:

$ cd /files
$ ls
Dockerfile  lambda_function.py	postgresql-11.19  postgresql-11.19.tar.gz  psycopg2-2.9.6  psycopg2-2.9.6.tar.gz  requirements.txt

Compiling Postgres

1. Install the dependencies required to build Postgres:

$ yum groupinstall "Development Tools"

2. Go into the folder containing the Postgres sources:

$ cd postgresql-11.19

3. Configure the build, compile and install Postgres:

$ ./configure --prefix `pwd` --without-readline --without-zlib
$ make
$ make install

4. If the compilation was successful, the binaries should be in the bin folder and you can check that it works by trying to run the pg_config command:

$ ./bin/pg_config

Compiling Psycopg2

1. Now go into the psycopg2 source directory and edit the setup.cfg file with the following (you may need to yum install vim to edit this file):

pg_config={path_to_postgresql_source/bin/pg_config}
static_libpq=1

2. Execute this to build the psycopg2 binary:

$ python setup.py build

3. The build artifacts are available in the build/lib.linux-x86_64-cpython-311/.

$ ls build/lib.linux-x86_64-cpython-311/
psycopg2

4. Copy the build/lib.linux-x86_64-cpython-311/psycopg2 folder into Postpone's backend/psycopg2.

Now Postpone should be ready to use the compiled psycopg2 package, as long as the Python version used by Postpone matches the one you just compiled.

References

• https://github.com/jkehler/awslambda-psycopg2#instructions-on-compiling-this-package-from-scratch

[thy-vk-dao]

@grantmcconnaughey hi, I followed your instructions to build Psycopg2 myself and the build was successful. But after building when importing I got this error:

>>> import psycopg2
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/app/psycopg2-2_9_1-Python_3.10/build/lib.linux-x86_64-cpython-310/psycopg2/__init__.py", line 51, in <module>
    from psycopg2._psycopg import (                     # noqa
ImportError: /app/psycopg2-2_9_1-Python_3.10/build/lib.linux-x86_64-cpython-310/psycopg2/_psycopg.cpython-310-x86_64-linux-gnu.so: undefined symbol: scram_SaltedPassword

[thy-vk-dao]

I have managed to make it work for me by downloading this version of Postgresql for the lambda image: v9.2.24 Then follow the rest of the build steps and it works!

[jkehler]

Python 3.11 support has just been merged in.