aws-cdk icon indicating copy to clipboard operation
aws-cdk copied to clipboard
verified publisher icon aws

(aws-lambda-python-alpha): Failed to install numpy 2.3.0 with Python 3.11 or lower

Open cedricfarinazzo opened this issue 10 months ago • 3 comments

Describe the bug

When creating a PythonFunction (using aws_lambda_python_alpha) with runtime Python 3.11, we cannot install the latest version of NumPy (2.3.0).

Regression Issue

  • [ ] Select this option if this issue appears to be a regression.

Last Known Working CDK Library Version

No response

Expected Behavior

Numpy 2.3.0 installs successfully inside the Lambda build container.

Current Behavior

$ cdk synth --app "python app.py"
#0 building with "builder" instance using docker driver

#1 [internal] load build definition from Dockerfile
#1 transferring dockerfile: 1.56kB 0.0s done
#1 DONE 0.0s

#2 [internal] load metadata for public.ecr.aws/sam/build-python3.11:latest
#2 DONE 0.0s

#3 [internal] load .dockerignore
#3 transferring context: 2B done
#3 DONE 0.0s

#4 [1/2] FROM public.ecr.aws/sam/build-python3.11:latest
#4 DONE 0.0s

#5 [2/2] RUN     python -m venv /usr/app/venv &&     mkdir /tmp/pip-cache &&     chmod -R 777 /tmp/pip-cache &&     pip install --upgrade pip &&     mkdir /tmp/poetry-cache &&     chmod -R 777 /tmp/poetry-cache &&     pip install pipenv==2022.4.8 poetry==1.5.1 &&     rm -rf /tmp/pip-cache/* /tmp/poetry-cache/*
#5 CACHED

#6 exporting to image
#6 exporting layers done
#6 writing image sha256:c2672077b57445177f3dd8ea27ebc99b20f9012a1f75e80fc665135ffff4d823 done
#6 naming to docker.io/library/cdk-1c75d961f4a9151bf4d2b0ae858f87b63fc0a44f5a603d6a8560e63a98d7e160 done
#6 DONE 0.0s
Bundling asset BrokenLambdaInstallStack/BrokenLambdaInstall/Code/Stage...
sending incremental file list
app.py
requirements.txt

sent 473 bytes  received 54 bytes  1,054.00 bytes/sec
total size is 291  speedup is 0.55
Collecting numpy==2.3.0 (from -r requirements.txt (line 1))
  Downloading numpy-2.3.0.tar.gz (20.4 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 20.4/20.4 MB 34.0 MB/s eta 0:00:00
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
  Installing backend dependencies: started
  Installing backend dependencies: finished with status 'done'
  Preparing metadata (pyproject.toml): started
  Preparing metadata (pyproject.toml): finished with status 'error'
  error: subprocess-exited-with-error
  
  × Preparing metadata (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> [19 lines of output]
      + /usr/app/venv/bin/python /tmp/pip-install-z68qfdn2/numpy_330e70ac49be48e98fa03b51a639488e/vendored-meson/meson/meson.py setup /tmp/pip-install-z68qfdn2/numpy_330e70ac49be48e98fa03b51a639488e /tmp/pip-install-z68qfdn2/numpy_330e70ac49be48e98fa03b51a639488e/.mesonpy-scy7d4n7 -Dbuildtype=release -Db_ndebug=if-release -Db_vscrt=md --native-file=/tmp/pip-install-z68qfdn2/numpy_330e70ac49be48e98fa03b51a639488e/.mesonpy-scy7d4n7/meson-python-native-file.ini
      The Meson build system
      Version: 1.6.1
      Source dir: /tmp/pip-install-z68qfdn2/numpy_330e70ac49be48e98fa03b51a639488e
      Build dir: /tmp/pip-install-z68qfdn2/numpy_330e70ac49be48e98fa03b51a639488e/.mesonpy-scy7d4n7
      Build type: native build
      Project name: NumPy
      Project version: 2.3.0
      C compiler for the host machine: cc (gcc 7.3.1 "cc (GCC) 7.3.1 20180712 (Red Hat 7.3.1-17)")
      C linker for the host machine: cc ld.bfd 2.29.1-31
      C++ compiler for the host machine: c++ (gcc 7.3.1 "c++ (GCC) 7.3.1 20180712 (Red Hat 7.3.1-17)")
      C++ linker for the host machine: c++ ld.bfd 2.29.1-31
      Cython compiler for the host machine: cython (cython 3.1.2)
      Host machine cpu family: x86_64
      Host machine cpu: x86_64

      ../meson.build:28:4: ERROR: Problem encountered: NumPy requires GCC >= 9.3

      A full log can be found at /tmp/pip-install-z68qfdn2/numpy_330e70ac49be48e98fa03b51a639488e/.mesonpy-scy7d4n7/meson-logs/meson-log.txt
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
error: metadata-generation-failed

× Encountered error while generating package metadata.
╰─> See above for output.

note: This is an issue with the package mentioned above, not pip.
hint: See above for details.
jsii.errors.JavaScriptError:
  Error: Failed to bundle asset BrokenLambdaInstallStack/BrokenLambdaInstall/Code/Stage, bundle output is located at /project/samples/broken_numpy/cdk.out/asset.2e8c0dc482fb5edd01a5045cf106b7f233d09a20ac4f1843ef7db24cf5e9244c-building: Error: docker exited with status 1
  --> Command: docker run --rm -u "1000:1000" -v "/project/samples/broken_numpy/example_lambda:/asset-input:delegated" -v "/project/samples/broken_numpy/cdk.out/asset.2e8c0dc482fb5edd01a5045cf106b7f233d09a20ac4f1843ef7db24cf5e9244c-building:/asset-output:delegated" -w "/asset-input" cdk-1c75d961f4a9151bf4d2b0ae858f87b63fc0a44f5a603d6a8560e63a98d7e160 bash -c "rsync -rLv /asset-input/ /asset-output && cd /asset-output && python -m pip install -r requirements.txt -t /asset-output"
      at AssetStaging.bundle (/tmp/jsii-kernel-kCXuHL/node_modules/aws-cdk-lib/core/lib/asset-staging.js:2:512)
      at AssetStaging.stageByBundling (/tmp/jsii-kernel-kCXuHL/node_modules/aws-cdk-lib/core/lib/asset-staging.js:1:5358)
      at stageThisAsset (/tmp/jsii-kernel-kCXuHL/node_modules/aws-cdk-lib/core/lib/asset-staging.js:1:2728)
      at Cache.obtain (/tmp/jsii-kernel-kCXuHL/node_modules/aws-cdk-lib/core/lib/private/cache.js:1:242)
      at new AssetStaging (/tmp/jsii-kernel-kCXuHL/node_modules/aws-cdk-lib/core/lib/asset-staging.js:1:3125)
      at new Asset (/tmp/jsii-kernel-kCXuHL/node_modules/aws-cdk-lib/aws-s3-assets/lib/asset.js:1:1252)
      at AssetCode.bind (/tmp/jsii-kernel-kCXuHL/node_modules/aws-cdk-lib/aws-lambda/lib/code.js:5:4375)
      at new Function2 (/tmp/jsii-kernel-kCXuHL/node_modules/aws-cdk-lib/aws-lambda/lib/function.js:1:11029)
      at new Function2 (/tmp/jsii-kernel-kCXuHL/node_modules/aws-cdk-lib/core/lib/prop-injectable.js:1:488)
      at new PythonFunction (/tmp/jsii-kernel-kCXuHL/node_modules/@aws-cdk/aws-lambda-python-alpha/lib/function.js:48:9)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/project/samples/broken_numpy/app.py", line 9, in <module>
    PythonFunction(
  File "/project/venv/lib/python3.11/site-packages/jsii/_runtime.py", line 118, in __call__
    inst = super(JSIIMeta, cast(JSIIMeta, cls)).__call__(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/project/venv/lib/python3.11/site-packages/aws_cdk/aws_lambda_python_alpha/__init__.py", line 1120, in __init__
    jsii.create(self.__class__, self, [scope, id, props])
  File "/project/venv/lib/python3.11/site-packages/jsii/_kernel/__init__.py", line 334, in create
    response = self.provider.create(
               ^^^^^^^^^^^^^^^^^^^^^
  File "/project/venv/lib/python3.11/site-packages/jsii/_kernel/providers/process.py", line 365, in create
    return self._process.send(request, CreateResponse)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/project/venv/lib/python3.11/site-packages/jsii/_kernel/providers/process.py", line 342, in send
    raise RuntimeError(resp.error) from JavaScriptError(resp.stack)
RuntimeError: Failed to bundle asset BrokenLambdaInstallStack/BrokenLambdaInstall/Code/Stage, bundle output is located at /project/samples/broken_numpy/cdk.out/asset.2e8c0dc482fb5edd01a5045cf106b7f233d09a20ac4f1843ef7db24cf5e9244c-building: Error: docker exited with status 1
--> Command: docker run --rm -u "1000:1000" -v "/project/samples/broken_numpy/example_lambda:/asset-input:delegated" -v "/project/samples/broken_numpy/cdk.out/asset.2e8c0dc482fb5edd01a5045cf106b7f233d09a20ac4f1843ef7db24cf5e9244c-building:/asset-output:delegated" -w "/asset-input" cdk-1c75d961f4a9151bf4d2b0ae858f87b63fc0a44f5a603d6a8560e63a98d7e160 bash -c "rsync -rLv /asset-input/ /asset-output && cd /asset-output && python -m pip install -r requirements.txt -t /asset-output"

Reproduction Steps

Create a stack with a 3.11 lambda using PythonFunction

from aws_cdk import App, Stack
from aws_cdk.aws_lambda_python_alpha import PythonFunction
from aws_cdk.aws_lambda import Runtime

app = App()

stack = Stack(app, "BrokenLambdaInstallStack")

PythonFunction(
    stack,
    "BrokenLambdaInstall",
    entry="example_lambda",
    index="app.py",
    runtime=Runtime.PYTHON_3_11,
)

app.synth()

Create the example_lambda folder with a requirements.txt with numpy==2.3.0

Possible Solution

AWS CDK uses the Dockerfile located at packages/@aws-cdk/aws-lambda-python-alpha/lib/Dockerfile to create the build environment for Lambda functions.

For Python 3.11, it uses the image public.ecr.aws/sam/build-python3.11:latest (see: https://github.com/aws/aws-sam-build-images/blob/develop/build-image-src/Dockerfile-python311), which is based on public.ecr.aws/lambda/python:3.11.

As mentioned in the NumPy issue numpy/numpy#29150, public.ecr.aws/lambda/python:3.11 is based on Amazon Linux 2, and NumPy no longer provides prebuilt wheels for it.

Simple test:

 docker run -it --rm --entrypoint /bin/bash public.ecr.aws/lambda/python:3.11                                                                                                                                                                                        1 ↵ 
bash-4.2# pip install numpy==2.3.0
Collecting numpy==2.3.0
  Downloading numpy-2.3.0.tar.gz (20.4 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 20.4/20.4 MB 44.1 MB/s eta 0:00:00
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Installing backend dependencies ... done
  Preparing metadata (pyproject.toml) ... error
  error: subprocess-exited-with-error

  × Preparing metadata (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> [20 lines of output]
      + /var/lang/bin/python3.11 /tmp/pip-install-eqtt3h5_/numpy_ba8a138bc9724d4fa8a1276690d4658d/vendored-meson/meson/meson.py setup /tmp/pip-install-eqtt3h5_/numpy_ba8a138bc9724d4fa8a1276690d4658d /tmp/pip-install-eqtt3h5_/numpy_ba8a138bc9724d4fa8a1276690d4658d/.mesonpy-s8svwir6 -Dbuildtype=release -Db_ndebug=if-release -Db_vscrt=md --native-file=/tmp/pip-install-eqtt3h5_/numpy_ba8a138bc9724d4fa8a1276690d4658d/.mesonpy-s8svwir6/meson-python-native-file.ini
      The Meson build system
      Version: 1.6.1
      Source dir: /tmp/pip-install-eqtt3h5_/numpy_ba8a138bc9724d4fa8a1276690d4658d
      Build dir: /tmp/pip-install-eqtt3h5_/numpy_ba8a138bc9724d4fa8a1276690d4658d/.mesonpy-s8svwir6
      Build type: native build
      Project name: NumPy
      Project version: 2.3.0

      ../meson.build:1:0: ERROR: Unknown compiler(s): [['cc'], ['gcc'], ['clang'], ['nvc'], ['pgcc'], ['icc'], ['icx']]
      The following exception(s) were encountered:
      Running `cc --version` gave "[Errno 2] No such file or directory: 'cc'"
      Running `gcc --version` gave "[Errno 2] No such file or directory: 'gcc'"
      Running `clang --version` gave "[Errno 2] No such file or directory: 'clang'"
      Running `nvc --version` gave "[Errno 2] No such file or directory: 'nvc'"
      Running `pgcc --version` gave "[Errno 2] No such file or directory: 'pgcc'"
      Running `icc --version` gave "[Errno 2] No such file or directory: 'icc'"
      Running `icx --version` gave "[Errno 2] No such file or directory: 'icx'"

      A full log can be found at /tmp/pip-install-eqtt3h5_/numpy_ba8a138bc9724d4fa8a1276690d4658d/.mesonpy-s8svwir6/meson-logs/meson-log.txt
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
error: metadata-generation-failed

× Encountered error while generating package metadata.
╰─> See above for output.

note: This is an issue with the package mentioned above, not pip.
hint: See above for details.

[notice] A new release of pip is available: 24.0 -> 25.1.1
[notice] To update, run: pip install --upgrade pip

Additional Information/Context

Related issue in numpy repo https://github.com/numpy/numpy/issues/29150

AWS CDK Library version (aws-cdk-lib)

2.200.1

AWS CDK CLI version

2.1018.0

Node.js Version

v22.16.01

OS

Debian12

Language

Python

Language Version

3.11

Other information

No response

cedricfarinazzo avatar Jun 11 '25 14:06 cedricfarinazzo

Hi @cedricfarinazzo,

Thank you for reporting this issue. You've identified a significant compatibility problem between NumPy 2.3.0 and the AWS Lambda Python build environment.

Root Cause: NumPy 2.3.0 introduced a breaking change requiring GCC >= 9.3, but the AWS Lambda Python 3.11 build environment (based on Amazon Linux 2) only has GCC 7.3.1. Additionally, NumPy no longer provides prebuilt wheels for Amazon Linux 2 as mentioned in numpy/numpy#29150.

The issue affects the entire CDK Python Lambda ecosystem because:

  • PythonFunction uses AWS SAM build images
  • These images are based on public.ecr.aws/lambda/python:3.11 (Amazon Linux 2)
  • The Dockerfile doesn't include additional build tools

Immediate Workarounds:

  1. Pin NumPy version: Use numpy<2.3.0 in your requirements.txt
  2. Use older NumPy: numpy==2.2.1 is the last version compatible with older GCC
  3. Custom build image: Override the build image with one containing newer build tools

Long-term Fix: This requires coordination between:

  • AWS Lambda team (updating base runtime images)
  • AWS SAM team (updating build images)
  • CDK team (potentially enhancing build process)

This is a service-level compatibility issue that will affect many users trying to use modern Python packages. While workaround is available, I am bringing it up to the team for visibility. Please help us prioritize with 👍 .

pahud avatar Jun 12 '25 09:06 pahud

Please help us prioritize with 👍 .

On the OP!

(That's what I assume will be used at least, e.g. by sorting issues, not reactions to the comment above.)

OJFord avatar Jun 13 '25 16:06 OJFord

Encountered today when installing packages into new pyenv environment with Python 3.11.11, rollback to numpy==2.2.1 worked.

AWS Cloud Desktop, Amazon Linux 2

oradomskyi avatar Jun 16 '25 14:06 oradomskyi

This issue has received a significant amount of attention so we are automatically upgrading its priority. A member of the community will see the re-prioritization and provide an update on the issue.

github-actions[bot] avatar Jun 29 '25 00:06 github-actions[bot]

Different package, same issue: contourpy>=1.3.3 (a dependency of matplotlib) cannot be installed in public.ecr.aws/lambda/python:3.11 because they no longer provide wheels compatible with Amazon Linux 2. Pinning contourpy==1.3.2 is a workaround.

WillMorrisonEnspi avatar Jul 30 '25 09:07 WillMorrisonEnspi

Not exactly. contourpy==1.3.3 can be installed with the standard OS gcc-c++ (yum install), if one installs numpy==2.2.6 before. The main issue with numpy is that the compiler in AL2 is too old.

StefanEnspired avatar Aug 01 '25 06:08 StefanEnspired

No inside knowledge but I'd be surprised if this gets fixed personally - 3.11 has 11 months left until deprecated on Lambda and that page already calls out this issue:

The Python 3.8-3.11 base images are based on the Amazon Linux 2 image. AL2023-based images provide several advantages over Amazon Linux 2, including a smaller deployment footprint and updated versions of libraries such as glibc.

I'd suggest seeking to upgrade to 3.13 (which is AL 2023) to resolve it, and that is what we did.

OJFord avatar Aug 01 '25 12:08 OJFord