Building functions that require native libraries to be present

[jasonmk]

Describe your idea/feature/enhancement

A number of the lambda languages have facilities for linking to C libraries for runtime performance. Ruby makes very heavy use of this, but it's not uncommon to see in Python or NodeJS as well. Even java is capable of it, though it's not all that common.

For example, in Ruby if you want to talk to a mysql database, you use a gem called mysql2 that expects to link against libmysqlclient.so. A layer is the natural place to store this .so file (rather than package it with your function code), but this doesn't solve the issue of having the development headers available at function build time.

Proposal

Provide a mechanism for injecting build-time dependencies into a build container as part of the sam build process so that native libraries can be compiled. Ideally this would also spit out a lambda layer with the runtime support libraries necessary for the function to actually run.

Things to consider:

1. This would likely require either an addition to the SAM spec or some type of buildspec file to specify what extra libraries are needed.
2. If we are generating a layer automatically, the transform (or possibly the CLI) would need to replace a ref to the lambda layer with the actual ARN of the dynamically created layer.

Additional Details

I've recorded my work on this and posted a Dockerfile and a couple scripts that I'm using to do this manually outside of the sam build process now. I don't consider this to be a final solution or even necessarily a valid approach. It does demonstrate part of the problem I'm facing though.

https://github.com/jasonmk/lambda-gem-compiler

[clin88]

Any idea when this will be prioritized? What's the expected cost of implementation?

This is a hard blocker for me. If I can get some guidance on how to implement this so that it's mergeable, I'd like to take a stab at it. Barring that, we'll likely work around this by forking SAM in a hacky way.

[jfuss]

@clin88 What are you trying to do? This issue is for building against native libraries in things like a Layer. If you ware just trying to build against the native libraries on Lambda you can use sam build --use-container.

[clin88]

Ya, building against native libraries is what I'm trying to solve. Namely, we use psycopg2cffi, which requires postgres, and pysaml2, which requires libxmlsec. And a few others I believe.

[mcouthon]

I also needed to do something like this (wanted to pip install mysqlclient), and was getting errors because mysql-devel wasn't installed in the image. A solution I could live with was updating the docker image to have the required packages. So I made one. sam build doesn't let you choose which docker image to use, to I renamed my image to be called lambci/lambda:build-python2.7, but something during sam build -u reverts to the original, empty image (not sure how/why).

[kornienkoanton]

@jfuss I heard this phrase many times If you ware just trying to build against the native libraries on Lambda you can use sam build --use-container, but I can't understand the easy thing, where I could specify the native package if inside of requirements.txt I've got python's packages. Where can I specify RUN yum install gcc-c++ pkgconfig poppler-cpp-devel python-devel redhat-rpm-config ?

[ryanash999]

@jfuss Our pipeline is already running as gitlab runner and i've been unsuccessful in using "--use-container" due to the docker in docker stuff. I just ran into another python module that has native C libraries and this issue has raised its ugly head.

[respire]

agree. This requirement should be satisfied via lambda layer.

https://docs.aws.amazon.com/lambda/latest/dg/configuration-layers.html#configuration-layers-path

it would be nice if sam-cli allows us to create a layer which includes common dependencies and ref that arn in template. so i can use docker container to build native extension and copy neccessary files to a local folder. and then let sam create the zip file, upload, create a layer and bind that layer to function.

[IaroslavR]

I also needed to do something like this (wanted to pip install mysqlclient), and was getting errors because mysql-devel wasn't installed in the image. A solution I could live with was updating the docker image to have the required packages. So I made one. sam build doesn't let you choose which docker image to use, to I renamed my image to be called lambci/lambda:build-python2.7, but something during sam build -u reverts to the original, empty image (not sure how/why).

Something like this can be used as workaround

[jondavidjohn]

@jfuss The problem here is that --use-container is not adequate because the container it is referencing does not have the native libraries present (in the case of postgres, the libpq-fe.h header).

Is there currently a story for adding system packages (like postgres packages) to our lambda containers via SAM? How would one add a dependency on a system-level package?

[jondavidjohn]

Is it possible to use a Dockerfile in conjunction with AWS SAM to augment the container used by a layer?

[zoellner]

This seems to be a breaking change from sam-cli v0.x to v1.x Previously lambci/lambda containers were used which provided yum to install additional packages. With the new amazon/aws-sam-cli-build-image-* containers that's not possible anymore

[pokhiii]

When I do sam local start-api and trigger my lambda it keeps giving:

Unable to get page count. Is poppler installed and in PATH?

on this line

from pdf2image import convert_from_bytes

pages = convert_from_bytes(pdf)

This, however, works when I run the same thing from the Python console. What can I do?

[sriram-mv]

Following up on this issue, AWS SAM supports Image based Lambda functions. You could define a base image which has all the requisites libraries and then build your serverless function based on that.

[jfuss]

Outside of moving to the PackageType: Image, for PackageType: Zip you can provide a customer build image through --build-image which will allow you to tell SAM CLI to use a specific image. You can build a custom build image and install the tools you directly need and then provide it to SAM CLI with sam build --use-container --build-image.

Closing this as there is now a way to get custom tools needed into the build image that can be linked against. If there is something still outstanding here, please create a new Github Issue and the team will take another look.

[github-actions[bot]]

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see. If you need more assistance, please either tag a team member or open a new issue that references this one. If you wish to keep having a conversation with other community members under this issue feel free to do so.