docker-lambda
docker-lambda copied to clipboard
Published
20 hours ago •
lambgeo
lambgeo
Failed building wheel for GDAL - missing `gcc` when building docker for Py3.11 + GDAL 3.6
Hello! Thank you for the project, it's really promising for my use case.
I'm trying the Advanced usage (https://github.com/lambgeo/docker-lambda?tab=readme-ov-file#2-advanced-need-other-dependencies) and I'm having the following error when building the image:
Building image…
Preparing build context archive…
[==================================================>]2/2 files
Done
Sending build context to Docker daemon…
[==================================================>] 616.0B
Done
Step 1/11 : FROM ghcr.io/lambgeo/lambda-gdal:3.6 as gdal
3.6: Pulling from lambgeo/lambda-gdal
Digest: sha256:a175fb80b3c2cda871d9fa36e93cec716734327a36ea06c14cdadc09178936c4
Status: Downloaded newer image for ghcr.io/lambgeo/lambda-gdal:3.6
---> c5879094d40e
Step 2/11 : FROM public.ecr.aws/lambda/python:3.11
---> d4f123181098
Step 3/11 : COPY --from=gdal /opt/lib/ /opt/lib/
---> bc97dcd894bf
Step 4/11 : COPY --from=gdal /opt/include/ /opt/include/
---> 0875bf50e97b
Step 5/11 : COPY --from=gdal /opt/share/ /opt/share/
---> 18ba81982b04
Step 6/11 : COPY --from=gdal /opt/bin/ /opt/bin/
---> 9cdd27c76bed
Step 7/11 : ENV GDAL_DATA=/opt/share/gdal PROJ_LIB=/opt/share/proj GDAL_CONFIG=/opt/bin/gdal-config GEOS_CONFIG=/opt/bin/geos-config PATH=/opt/bin:$PATH
---> Running in e9cfd730d78d
Removing intermediate container e9cfd730d78d
---> 72fd2520b5e5
Step 8/11 : ENV PACKAGE_PREFIX=/var/task
---> Running in 32dc43f0ad48
Removing intermediate container 32dc43f0ad48
---> dc04248e2c05
Step 9/11 : COPY handler.py ${PACKAGE_PREFIX}/handler.py
---> cf9879a5b079
Step 10/11 : RUN python -m pip install GDAL==$(gdal-config --version) -t $PACKAGE_PREFIX
---> Running in a64cb1e7b343
Collecting GDAL==3.6.4
Downloading GDAL-3.6.4.tar.gz (758 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 758.1/758.1 kB 4.0 MB/s eta 0:00:00
Installing build dependencies: started
Installing build dependencies: finished with status 'done'
Getting requirements to build wheel: started
Getting requirements to build wheel: finished with status 'done'
Preparing metadata (pyproject.toml): started
Preparing metadata (pyproject.toml): finished with status 'done'
Building wheels for collected packages: GDAL
Building wheel for GDAL (pyproject.toml): started
Building wheel for GDAL (pyproject.toml): finished with status 'error'
error: subprocess-exited-with-error
× Building wheel for GDAL (pyproject.toml) did not run successfully.
│ exit code: 1
╰─> [126 lines of output]
WARNING: numpy not available! Array support will not be enabled
running bdist_wheel
running build
running build_py
creating build
creating build/lib.linux-aarch64-cpython-311
creating build/lib.linux-aarch64-cpython-311/osgeo
copying osgeo/gdal.py -> build/lib.linux-aarch64-cpython-311/osgeo
copying osgeo/gnm.py -> build/lib.linux-aarch64-cpython-311/osgeo
copying osgeo/gdalnumeric.py -> build/lib.linux-aarch64-cpython-311/osgeo
copying osgeo/ogr.py -> build/lib.linux-aarch64-cpython-311/osgeo
copying osgeo/osr.py -> build/lib.linux-aarch64-cpython-311/osgeo
copying osgeo/gdalconst.py -> build/lib.linux-aarch64-cpython-311/osgeo
copying osgeo/gdal_array.py -> build/lib.linux-aarch64-cpython-311/osgeo
copying osgeo/__init__.py -> build/lib.linux-aarch64-cpython-311/osgeo
creating build/lib.linux-aarch64-cpython-311/osgeo_utils
copying gdal-utils/osgeo_utils/gdal2tiles.py -> build/lib.linux-aarch64-cpython-311/osgeo_utils
copying gdal-utils/osgeo_utils/gdal_merge.py -> build/lib.linux-aarch64-cpython-311/osgeo_utils
copying gdal-utils/osgeo_utils/ogrmerge.py -> build/lib.linux-aarch64-cpython-311/osgeo_utils
copying gdal-utils/osgeo_utils/rgb2pct.py -> build/lib.linux-aarch64-cpython-311/osgeo_utils
copying gdal-utils/osgeo_utils/gdal_polygonize.py -> build/lib.linux-aarch64-cpython-311/osgeo_utils
copying gdal-utils/osgeo_utils/gdalattachpct.py -> build/lib.linux-aarch64-cpython-311/osgeo_utils
copying gdal-utils/osgeo_utils/ogr_layer_algebra.py -> build/lib.linux-aarch64-cpython-311/osgeo_utils
copying gdal-utils/osgeo_utils/gdal_edit.py -> build/lib.linux-aarch64-cpython-311/osgeo_utils
copying gdal-utils/osgeo_utils/gdal_proximity.py -> build/lib.linux-aarch64-cpython-311/osgeo_utils
copying gdal-utils/osgeo_utils/gdalcompare.py -> build/lib.linux-aarch64-cpython-311/osgeo_utils
copying gdal-utils/osgeo_utils/gdal2xyz.py -> build/lib.linux-aarch64-cpython-311/osgeo_utils
copying gdal-utils/osgeo_utils/gdal_calc.py -> build/lib.linux-aarch64-cpython-311/osgeo_utils
copying gdal-utils/osgeo_utils/pct2rgb.py -> build/lib.linux-aarch64-cpython-311/osgeo_utils
copying gdal-utils/osgeo_utils/gdal_fillnodata.py -> build/lib.linux-aarch64-cpython-311/osgeo_utils
copying gdal-utils/osgeo_utils/gdal_pansharpen.py -> build/lib.linux-aarch64-cpython-311/osgeo_utils
copying gdal-utils/osgeo_utils/gdal_retile.py -> build/lib.linux-aarch64-cpython-311/osgeo_utils
copying gdal-utils/osgeo_utils/__init__.py -> build/lib.linux-aarch64-cpython-311/osgeo_utils
copying gdal-utils/osgeo_utils/gdal_sieve.py -> build/lib.linux-aarch64-cpython-311/osgeo_utils
copying gdal-utils/osgeo_utils/gdalmove.py -> build/lib.linux-aarch64-cpython-311/osgeo_utils
creating build/lib.linux-aarch64-cpython-311/osgeo_utils/auxiliary
copying gdal-utils/osgeo_utils/auxiliary/numpy_util.py -> build/lib.linux-aarch64-cpython-311/osgeo_utils/auxiliary
copying gdal-utils/osgeo_utils/auxiliary/progress.py -> build/lib.linux-aarch64-cpython-311/osgeo_utils/auxiliary
copying gdal-utils/osgeo_utils/auxiliary/util.py -> build/lib.linux-aarch64-cpython-311/osgeo_utils/auxiliary
copying gdal-utils/osgeo_utils/auxiliary/osr_util.py -> build/lib.linux-aarch64-cpython-311/osgeo_utils/auxiliary
copying gdal-utils/osgeo_utils/auxiliary/extent_util.py -> build/lib.linux-aarch64-cpython-311/osgeo_utils/auxiliary
copying gdal-utils/osgeo_utils/auxiliary/gdal_argparse.py -> build/lib.linux-aarch64-cpython-311/osgeo_utils/auxiliary
copying gdal-utils/osgeo_utils/auxiliary/color_palette.py -> build/lib.linux-aarch64-cpython-311/osgeo_utils/auxiliary
copying gdal-utils/osgeo_utils/auxiliary/array_util.py -> build/lib.linux-aarch64-cpython-311/osgeo_utils/auxiliary
copying gdal-utils/osgeo_utils/auxiliary/rectangle.py -> build/lib.linux-aarch64-cpython-311/osgeo_utils/auxiliary
copying gdal-utils/osgeo_utils/auxiliary/raster_creation.py -> build/lib.linux-aarch64-cpython-311/osgeo_utils/auxiliary
copying gdal-utils/osgeo_utils/auxiliary/batch_creator.py -> build/lib.linux-aarch64-cpython-311/osgeo_utils/auxiliary
copying gdal-utils/osgeo_utils/auxiliary/color_table.py -> build/lib.linux-aarch64-cpython-311/osgeo_utils/auxiliary
copying gdal-utils/osgeo_utils/auxiliary/base.py -> build/lib.linux-aarch64-cpython-311/osgeo_utils/auxiliary
copying gdal-utils/osgeo_utils/auxiliary/__init__.py -> build/lib.linux-aarch64-cpython-311/osgeo_utils/auxiliary
creating build/lib.linux-aarch64-cpython-311/osgeo_utils/samples
copying gdal-utils/osgeo_utils/samples/gdal_remove_towgs84.py -> build/lib.linux-aarch64-cpython-311/osgeo_utils/samples
copying gdal-utils/osgeo_utils/samples/densify.py -> build/lib.linux-aarch64-cpython-311/osgeo_utils/samples
copying gdal-utils/osgeo_utils/samples/gdal_rmdir.py -> build/lib.linux-aarch64-cpython-311/osgeo_utils/samples
copying gdal-utils/osgeo_utils/samples/vec_tr_spat.py -> build/lib.linux-aarch64-cpython-311/osgeo_utils/samples
copying gdal-utils/osgeo_utils/samples/gdal_create_pdf.py -> build/lib.linux-aarch64-cpython-311/osgeo_utils/samples
copying gdal-utils/osgeo_utils/samples/gdalident.py -> build/lib.linux-aarch64-cpython-311/osgeo_utils/samples
copying gdal-utils/osgeo_utils/samples/fix_gpkg.py -> build/lib.linux-aarch64-cpython-311/osgeo_utils/samples
copying gdal-utils/osgeo_utils/samples/crs2crs2grid.py -> build/lib.linux-aarch64-cpython-311/osgeo_utils/samples
copying gdal-utils/osgeo_utils/samples/tolatlong.py -> build/lib.linux-aarch64-cpython-311/osgeo_utils/samples
copying gdal-utils/osgeo_utils/samples/gcps2wld.py -> build/lib.linux-aarch64-cpython-311/osgeo_utils/samples
copying gdal-utils/osgeo_utils/samples/gcps2ogr.py -> build/lib.linux-aarch64-cpython-311/osgeo_utils/samples
copying gdal-utils/osgeo_utils/samples/epsg_tr.py -> build/lib.linux-aarch64-cpython-311/osgeo_utils/samples
copying gdal-utils/osgeo_utils/samples/ogr_dispatch.py -> build/lib.linux-aarch64-cpython-311/osgeo_utils/samples
copying gdal-utils/osgeo_utils/samples/gdal_vrtmerge.py -> build/lib.linux-aarch64-cpython-311/osgeo_utils/samples
copying gdal-utils/osgeo_utils/samples/loslas2ntv2.py -> build/lib.linux-aarch64-cpython-311/osgeo_utils/samples
copying gdal-utils/osgeo_utils/samples/hsv_merge.py -> build/lib.linux-aarch64-cpython-311/osgeo_utils/samples
copying gdal-utils/osgeo_utils/samples/load2odbc.py -> build/lib.linux-aarch64-cpython-311/osgeo_utils/samples
copying gdal-utils/osgeo_utils/samples/gdal_rm.py -> build/lib.linux-aarch64-cpython-311/osgeo_utils/samples
copying gdal-utils/osgeo_utils/samples/fft.py -> build/lib.linux-aarch64-cpython-311/osgeo_utils/samples
copying gdal-utils/osgeo_utils/samples/ogrupdate.py -> build/lib.linux-aarch64-cpython-311/osgeo_utils/samples
copying gdal-utils/osgeo_utils/samples/gdal_auth.py -> build/lib.linux-aarch64-cpython-311/osgeo_utils/samples
copying gdal-utils/osgeo_utils/samples/wcs_virtds_params.py -> build/lib.linux-aarch64-cpython-311/osgeo_utils/samples
copying gdal-utils/osgeo_utils/samples/build_jp2_from_xml.py -> build/lib.linux-aarch64-cpython-311/osgeo_utils/samples
copying gdal-utils/osgeo_utils/samples/gdal_mkdir.py -> build/lib.linux-aarch64-cpython-311/osgeo_utils/samples
copying gdal-utils/osgeo_utils/samples/gdalfilter.py -> build/lib.linux-aarch64-cpython-311/osgeo_utils/samples
copying gdal-utils/osgeo_utils/samples/gdal_lut.py -> build/lib.linux-aarch64-cpython-311/osgeo_utils/samples
copying gdal-utils/osgeo_utils/samples/gdal2grd.py -> build/lib.linux-aarch64-cpython-311/osgeo_utils/samples
copying gdal-utils/osgeo_utils/samples/vec_tr.py -> build/lib.linux-aarch64-cpython-311/osgeo_utils/samples
copying gdal-utils/osgeo_utils/samples/esri2wkt.py -> build/lib.linux-aarch64-cpython-311/osgeo_utils/samples
copying gdal-utils/osgeo_utils/samples/gdalchksum.py -> build/lib.linux-aarch64-cpython-311/osgeo_utils/samples
copying gdal-utils/osgeo_utils/samples/dump_jp2.py -> build/lib.linux-aarch64-cpython-311/osgeo_utils/samples
copying gdal-utils/osgeo_utils/samples/gdal_cp.py -> build/lib.linux-aarch64-cpython-311/osgeo_utils/samples
copying gdal-utils/osgeo_utils/samples/gdal_zip.py -> build/lib.linux-aarch64-cpython-311/osgeo_utils/samples
copying gdal-utils/osgeo_utils/samples/histrep.py -> build/lib.linux-aarch64-cpython-311/osgeo_utils/samples
copying gdal-utils/osgeo_utils/samples/validate_cloud_optimized_geotiff.py -> build/lib.linux-aarch64-cpython-311/osgeo_utils/samples
copying gdal-utils/osgeo_utils/samples/gdalimport.py -> build/lib.linux-aarch64-cpython-311/osgeo_utils/samples
copying gdal-utils/osgeo_utils/samples/rel.py -> build/lib.linux-aarch64-cpython-311/osgeo_utils/samples
copying gdal-utils/osgeo_utils/samples/ogr_build_junction_table.py -> build/lib.linux-aarch64-cpython-311/osgeo_utils/samples
copying gdal-utils/osgeo_utils/samples/make_fuzzer_friendly_archive.py -> build/lib.linux-aarch64-cpython-311/osgeo_utils/samples
copying gdal-utils/osgeo_utils/samples/get_soundg.py -> build/lib.linux-aarch64-cpython-311/osgeo_utils/samples
copying gdal-utils/osgeo_utils/samples/gcps2vec.py -> build/lib.linux-aarch64-cpython-311/osgeo_utils/samples
copying gdal-utils/osgeo_utils/samples/gdallocationinfo.py -> build/lib.linux-aarch64-cpython-311/osgeo_utils/samples
copying gdal-utils/osgeo_utils/samples/tigerpoly.py -> build/lib.linux-aarch64-cpython-311/osgeo_utils/samples
copying gdal-utils/osgeo_utils/samples/tile_extent_from_raster.py -> build/lib.linux-aarch64-cpython-311/osgeo_utils/samples
copying gdal-utils/osgeo_utils/samples/classify.py -> build/lib.linux-aarch64-cpython-311/osgeo_utils/samples
copying gdal-utils/osgeo_utils/samples/assemblepoly.py -> build/lib.linux-aarch64-cpython-311/osgeo_utils/samples
copying gdal-utils/osgeo_utils/samples/val_repl.py -> build/lib.linux-aarch64-cpython-311/osgeo_utils/samples
copying gdal-utils/osgeo_utils/samples/validate_gpkg.py -> build/lib.linux-aarch64-cpython-311/osgeo_utils/samples
copying gdal-utils/osgeo_utils/samples/mkgraticule.py -> build/lib.linux-aarch64-cpython-311/osgeo_utils/samples
copying gdal-utils/osgeo_utils/samples/ogr2ogr.py -> build/lib.linux-aarch64-cpython-311/osgeo_utils/samples
copying gdal-utils/osgeo_utils/samples/gdalcopyproj.py -> build/lib.linux-aarch64-cpython-311/osgeo_utils/samples
copying gdal-utils/osgeo_utils/samples/jpeg_in_tiff_extract.py -> build/lib.linux-aarch64-cpython-311/osgeo_utils/samples
copying gdal-utils/osgeo_utils/samples/ogr2vrt.py -> build/lib.linux-aarch64-cpython-311/osgeo_utils/samples
copying gdal-utils/osgeo_utils/samples/gdal_ls.py -> build/lib.linux-aarch64-cpython-311/osgeo_utils/samples
copying gdal-utils/osgeo_utils/samples/ogrinfo.py -> build/lib.linux-aarch64-cpython-311/osgeo_utils/samples
copying gdal-utils/osgeo_utils/samples/magphase.py -> build/lib.linux-aarch64-cpython-311/osgeo_utils/samples
copying gdal-utils/osgeo_utils/samples/gdalinfo.py -> build/lib.linux-aarch64-cpython-311/osgeo_utils/samples
copying gdal-utils/osgeo_utils/samples/validate_jp2.py -> build/lib.linux-aarch64-cpython-311/osgeo_utils/samples
copying gdal-utils/osgeo_utils/samples/__init__.py -> build/lib.linux-aarch64-cpython-311/osgeo_utils/samples
running build_ext
gcc -pthread -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -I/var/lang/include/python3.11 -I/opt/include -c gdal_python_cxx11_test.cpp -o gdal_python_cxx11_test.o
gcc -pthread -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -I/var/lang/include/python3.11 -I/opt/include -c gdal_python_cxx11_test.cpp -o gdal_python_cxx11_test.o -std=c++11
building 'osgeo._gdal' extension
creating build/temp.linux-aarch64-cpython-311
building 'osgeo._gdalconst' extension
creating build/temp.linux-aarch64-cpython-311/extensions
gcc -pthread -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -I/var/lang/include/python3.11 -I/opt/include -c extensions/gdalconst_wrap.c -o build/temp.linux-aarch64-cpython-311/extensions/gdalconst_wrap.o -I/opt/include
gcc -pthread -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -I/var/lang/include/python3.11 -I/opt/include -c extensions/gdal_wrap.cpp -o build/temp.linux-aarch64-cpython-311/extensions/gdal_wrap.o -I/opt/include
building 'osgeo._osr' extension
gcc -pthread -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -I/var/lang/include/python3.11 -I/opt/include -c extensions/osr_wrap.cpp -o build/temp.linux-aarch64-cpython-311/extensions/osr_wrap.o -I/opt/include
building 'osgeo._ogr' extension
gcc -pthread -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -I/var/lang/include/python3.11 -I/opt/include -c extensions/ogr_wrap.cpp -o build/temp.linux-aarch64-cpython-311/extensions/ogr_wrap.o -I/opt/include
building 'osgeo._gnm' extension
gcc -pthread -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -I/var/lang/include/python3.11 -I/opt/include -c extensions/gnm_wrap.cpp -o build/temp.linux-aarch64-cpython-311/extensions/gnm_wrap.o -I/opt/include
error: command 'gcc' failed: No such file or directory
[end of output]
note: This error originates from a subprocess, and is likely not a problem with pip.
ERROR: Failed building wheel for GDAL
ERROR: Could not build wheels for GDAL, which is required to install pyproject.toml-based projects
Failed to build GDAL
[notice] A new release of pip is available: 23.2.1 -> 24.0
[notice] To update, run: pip install --upgrade pip
Error response from daemon: The command '/bin/sh -c python -m pip install GDAL==$(gdal-config --version) -t $PACKAGE_PREFIX' returned a non-zero code: 1
I tried both with a MacOS system (M1 with Ventura 13) and a Linux (Intel x64 with Ubuntu 22). I'm using Python 3.11 and GDAL 3.6 , so my Dockerfile is:
FROM ghcr.io/lambgeo/lambda-gdal:3.6 as gdal
# This example assume that you are creating a lambda package for python 3.11
FROM public.ecr.aws/lambda/python:3.11
# Bring C libs from lambgeo/lambda-gdal image
COPY --from=gdal /opt/lib/ /opt/lib/
COPY --from=gdal /opt/include/ /opt/include/
COPY --from=gdal /opt/share/ /opt/share/
COPY --from=gdal /opt/bin/ /opt/bin/
ENV \
GDAL_DATA=/opt/share/gdal \
PROJ_LIB=/opt/share/proj \
GDAL_CONFIG=/opt/bin/gdal-config \
GEOS_CONFIG=/opt/bin/geos-config \
PATH=/opt/bin:$PATH
ENV PACKAGE_PREFIX=/var/task
# Copy local files
COPY handler.py ${PACKAGE_PREFIX}/handler.py
# install package
# This example shows how to install GDAL python bindings for gdal 3.6
# The GDAL version should be the same as the one provided by the `lambgeo/lambda-gdal` image
RUN python -m pip install GDAL==$(gdal-config --version) -t $PACKAGE_PREFIX
# Create package.zip
RUN cd $PACKAGE_PREFIX && zip -r9q /tmp/package.zip *
Thank you in advance, let me know how I can help!
Edit: when commenting out that step, it also fails in the zipping step:
Building image…
Preparing build context archive…
[==================================================>]2/2 files
Done
Sending build context to Docker daemon…
[==================================================>] 620.0B
Done
Step 1/10 : FROM ghcr.io/lambgeo/lambda-gdal:3.6 as gdal
---> c5879094d40e
Step 2/10 : FROM public.ecr.aws/lambda/python:3.11
---> d4f123181098
Step 3/10 : COPY --from=gdal /opt/lib/ /opt/lib/
---> Using cache
---> bc97dcd894bf
Step 4/10 : COPY --from=gdal /opt/include/ /opt/include/
---> Using cache
---> 0875bf50e97b
Step 5/10 : COPY --from=gdal /opt/share/ /opt/share/
---> Using cache
---> 18ba81982b04
Step 6/10 : COPY --from=gdal /opt/bin/ /opt/bin/
---> Using cache
---> 9cdd27c76bed
Step 7/10 : ENV GDAL_DATA=/opt/share/gdal PROJ_LIB=/opt/share/proj GDAL_CONFIG=/opt/bin/gdal-config GEOS_CONFIG=/opt/bin/geos-config PATH=/opt/bin:$PATH
---> Using cache
---> 72fd2520b5e5
Step 8/10 : ENV PACKAGE_PREFIX=/var/task
---> Using cache
---> dc04248e2c05
Step 9/10 : COPY handler.py ${PACKAGE_PREFIX}/handler.py
---> Using cache
---> cf9879a5b079
Step 10/10 : RUN cd $PACKAGE_PREFIX && zip -r9q /tmp/package.zip *
---> Running in 039896078d67
/bin/sh: zip: command not found
Error response from daemon: The command '/bin/sh -c cd $PACKAGE_PREFIX && zip -r9q /tmp/package.zip *' returned a non-zero code: 127
can you try adding
RUN yum clean all && yum -y update && yum -y install gcc gcc-c++ unzip curl zip
before the pip install step
