Guidance: how can I include psycopg2 in the layer?

[takeda]

I need to be able to include psycopg2 as an dependency to my project. Unfortunately I can't use psycopg2-binary or aws-psycopg2 and similar libraries.

When I build requirements surprisingly it looks like code compiles, but when it is actually run it crashes with message that there's no libpq.so.5. I'm surprised that it works as in the container that's used for building that library is not present.

Looked at some PRs and documentation looks like I can include arbitrary files as well as modify the container.

These settings I'm trying:

  pythonRequirements:
    #dockerImage: mlupin/docker-lambda:python3.9-build
    dockerizePip: non-linux
    layer: true
    useDownloadCache: true
    useStaticCache: true
    cacheLocation: .serverless-cache
    staticCacheMaxVersions: 3
    #dockerImage: mlupin/docker-lambda:python3.9-build
    dockerFile: Dockerfile-dependencies
    dockerExtraFiles:
      - /usr/lib64/libpq.so.5

the Dockerfile-dependencies looks like this:

FROM lambci/lambda:build-python3.8
RUN yum install -y postgresql-libs

When I download the installed lambda it doesn't contains libpq.so.5 anywhere.

Am I doing things right?

[eschwartz]

Did you have any luck with this? I'm still stuck trying to get pyscopg running in lambda.

It works when deployed via github actions, but not when I deploy locally on my Mac M3. I assume it's because it's being compiled for the wrong platform.

[takeda]

No, I had to make psycopg2-binary work which I wanted to avoid. This feels like either this functionality is broken or documentation is lacking.

[pgrzesik]

Hey @eschwartz 👋 It seems to be an issue with mismatching architectures between build environment and runtime on AWS Lambda. I wrote an article on how to tackle that for NumPy, but it should be applicable for other similar packages like psycopg too. Let me know if that helps

Link: https://pgrzesik.com/posts/making-numpy-serverless/