Maintenance: Create a Powertools Lambda layer for each Python version

[leandrodamascena]

Why is this needed?

Powertools is growing very fast and we are adding more features day by day. Some of these features require compiled libraries and those libraries must be compiled for the specific version of Python. An example of this is aws-encryption-sdk which brings cryptography as a dependency. We had an issue (https://github.com/aws-powertools/powertools-lambda-python/issues/3852) recently and we need to implement it to avoid more problems in the future.

Which area does this relate to?

Other

Solution

Build the Powertools Lambda layer for the specific version of Python.

Acknowledgment

• [X] This request meets Powertools for AWS Lambda (Python) Tenets
• [ ] Should this be considered in other Powertools for AWS Lambda languages? i.e. Java, TypeScript, and .NET

[heitorlessa]

One small note to also increase our canary testing across all Python versions

[leandrodamascena]

Assigning to @sthulb!

[leandrodamascena]

This is the main blocker preventing us from releasing Powertools v3, so I'm taking over this to release v3 as soon as possible. I'm working on this since last Friday and I've made some progress.

I can now build the Powertools layer using the current five supported Python versions, but I still have some issues we need to work together to get them passed:

1 - We are currently building the Powertools layer using Python 3.8 and need make this flexible to accept any version of python. We need to pay attention to Python 3.12 because it is using Amazon Linux 2023 instead of AL2, which may require us to create a specific Dockerfile for this. Maybe using arguments, maybe using multistage build, idk, need to test. Can you help with this item @am29d?

2 - We are currently using Buildx + QEMU to emulate arch arm64 on x86 servers, but it looks like we need to improve things here to make it work properly. If we look at this line we are uploading the file twice with the same name, which forces a replacement of the file and is ALWAYS using the arm64. Fortunately, our current layer does not have any pre-compiled libraries and therefore works correctly. Am I wrong @heitorlessa?

3 - I need help thinking about how to show all these new ARNs on our documentation. We also need to think about GovCloud layers.

I think with some help on a few items we will unblock this item soon and we will move towards the V3 release.

Thanks

[dreamorosi]

3 - I need help thinking about how to show all these new ARNs on our documentation. We also need to think about GovCloud layers.

Today we have this:

which is two tabs one for x86 and one for ARM, with each one containing the ARNs for each region.

In my opinion, Gov Cloud could just be another entry in that list, just like any other region.

Regarding the others, before making a suggestion - it'd be helpful for me to have an example of the ARNs we'll have after making this change.

[leandrodamascena]

Python 3.12 is failing when using AWS based images due to using AL2023. I know how to solve it, but I'll leave it until the end

Dockerfile:17
--------------------
path /tmp/jsii-kernel-Rx5134/node_modules/cdk-aws-lambda-powertools-layer/layer/Python
  15 |     # PACKAGE_SUFFIX = ''
  16 |     
  17 | >>> RUN yum update -y && yum install -y zip unzip wget tar gzip binutils
  18 |     
  19 |     # Install build essentials
--------------------
ERROR: failed to solve: process "/bin/sh -c yum update -y && yum install -y zip unzip wget tar gzip binutils" did not complete successfully: exit code: 127
jsii.errors.JavaScriptError: 
  Error: docker exited with status 1
  --> Command: docker build -t cdk-ff22dfe9b0c5c12f271427156b61df72c8c53b32355ffe9b809b982ecc7b23ec --platform "linux/amd64" --build-arg "PACKAGE_SUFFIX=[all]==1.0.0" --build-arg "PYTHON_VERSION=3.12" "/tmp/jsii-kernel-Rx5134/node_modules/cdk-aws-lambda-powertools-layer/layer/Python"
      at dockerExec (/tmp/jsii-kernel-Rx5134/node_modules/aws-cdk-lib/core/lib/private/asset-staging.js:2:237)
      at DockerImage.fromBuild (/tmp/jsii-kernel-Rx5134/node_modules/aws-cdk-lib/core/lib/bundling.js:1:4761)
      at Code.fromDockerBuild (/tmp/jsii-kernel-Rx5134/node_modules/aws-cdk-lib/aws-lambda/lib/code.js:5:531)
      at new LambdaPowertoolsLayer (/tmp/jsii-kernel-Rx5134/node_modules/cdk-aws-lambda-powertools-layer/lib/lambda-powertools-layer.js:59:49)
      at Kernel._Kernel_create (/tmp/tmpxfbwozhn/lib/program.js:10119:25)
      at Kernel.create (/tmp/tmpxfbwozhn/lib/program.js:9790:93)
      at KernelHost.processRequest (/tmp/tmpxfbwozhn/lib/program.js:11707:36)
      at KernelHost.run (/tmp/tmpxfbwozhn/lib/program.js:11667:22)
      at Immediate._onImmediate (/tmp/tmpxfbwozhn/lib/program.js:11668:46)
      at process.processImmediate (node:internal/timers:476:21)

[leandrodamascena]

Problem with Python 3.12 fixed! We are now building the CDK on the five current supported python versions.

Next step: deploy a layer using the pipeline in my personal account

[leandrodamascena]

Some layers are failing to deploy, but they are simple to resolve. We already have an almost complete pipeline working 🚀

[leandrodamascena]

Work in progress!

[leandrodamascena]

Closed via #4826

[github-actions[bot]]

⚠️COMMENT VISIBILITY WARNING⚠️

This issue is now closed. Please be mindful that future comments are hard for our team to see.

If you need more assistance, please either tag a team member or open a new issue that references this one.

If you wish to keep having a conversation with other community members under this issue feel free to do so.

[github-actions[bot]]

This is now released under 2.43.0 version!