Cloud Run Buildpacks error hiredis

[cychongaspevo]

Encountered error while trying to install package. Basically same build 3 days ago, and today unable to build this build...

Failed to build hiredis
Installing collected packages: wrapt, wcwidth, starkbank-ecdsa, pytz, pyasn1, msgpack, incremental, dj-database-url, constantly, certifi, billiard, vine, urllib3, uritemplate, txaio, sqlparse, sorl-thumbnail, smart-open, six, setuptools, rsa, python-http-client, python-dotenv, pyparsing, pycparser, pyasn1-modules, psycopg2-binary, prompt-toolkit, Pillow, oauthlib, jmespath, install, idna, hiredis, django-environ, Deprecated, click, chardet, cachetools, attrs, async-timeout, asgiref, zope.interface, sendgrid, requests, python-dateutil, protobuf, packaging, hyperlink, httplib2, gunicorn, grpcio, google-auth, Django, click-repl, click-plugins, click-didyoumean, cffi, Automat, amqp, aioredis, Twisted, python-crontab, proto-plus, kombu, googleapis-common-protos, google-crc32c, google-auth-httplib2, djangorestframework, django-timezone-field, django-storages, django-cors-headers, cryptography, CacheControl, botocore, service-identity, s3transfer, pyOpenSSL, jwcrypto, google-resumable-media, google-api-core, celery, autobahn, google-cloud-core, google-api-python-client, django-oauth-toolkit, django-celery-beat, boto3, google-cloud-storage, google-cloud-firestore, google-cloud-bigquery, daphne, firebase-admin, channels, fcm-django, channels-redis
  Running setup.py install for hiredis: finished with status 'error'
  error: subprocess-exited-with-error
  
  × Running setup.py install for hiredis did not run successfully.
  │ exit code: 1
  ╰─> [29 lines of output]
      /tmp/pip-install-fz0508fi/hiredis_5131b8c3e7584387a9e090f406cf4c84/setup.py:7: DeprecationWarning: the imp module is deprecated in favour of importlib and slated for removal in Python 3.12; see the module's documentation for alternative uses
        import sys, imp, os, glob, io
      /layers/google.python.pip/pip/lib/python3.10/site-packages/setuptools/dist.py:771: UserWarning: Usage of dash-separated 'description-file' will not be supported in future versions. Please use the underscore name 'description_file' instead
        warnings.warn(
      running install
      /layers/google.python.pip/pip/lib/python3.10/site-packages/setuptools/command/install.py:34: SetuptoolsDeprecationWarning: setup.py install is deprecated. Use build and pip and other standards-based tools.
        warnings.warn(
      running build
      running build_py
      creating build
      creating build/lib.linux-x86_64-cpython-310
      creating build/lib.linux-x86_64-cpython-310/hiredis
      copying hiredis/version.py -> build/lib.linux-x86_64-cpython-310/hiredis
      copying hiredis/__init__.py -> build/lib.linux-x86_64-cpython-310/hiredis
      copying hiredis/hiredis.pyi -> build/lib.linux-x86_64-cpython-310/hiredis
      copying hiredis/py.typed -> build/lib.linux-x86_64-cpython-310/hiredis
      running build_ext
      building 'hiredis.hiredis' extension
      creating build/temp.linux-x86_64-cpython-310
      creating build/temp.linux-x86_64-cpython-310/src
      creating build/temp.linux-x86_64-cpython-310/vendor
      creating build/temp.linux-x86_64-cpython-310/vendor/hiredis
      x86_64-linux-gnu-gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fstack-protector-strong -g -Wformat -Werror=format-security -fPIC -Ivendor -I/opt/python/include/python3.10 -c src/hiredis.c -o build/temp.linux-x86_64-cpython-310/src/hiredis.o
      In file included from src/hiredis.c:1:0:
      src/hiredis.h:4:10: fatal error: Python.h: No such file or directory
       #include <Python.h>
                ^~~~~~~~~~
      compilation terminated.
      error: command '/usr/bin/x86_64-linux-gnu-gcc' failed with exit code 1
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
error: legacy-install-failure

× Encountered error while trying to install package.
╰─> hiredis

[rajjagesar]

Similar problem here but with the psycopg dependency on a python runtime. Build worked a couple of days ago, no changes.

I can confirm from cloud build logs that the required OS dependencies such as libpq-dev and python3-dev are correctly installed in the custom builder and runner image. I guess maybe another specific OS dependency in the base image has suddenly gone missing or the aforementioned custom images are not correctly picked up by packer, thus causing the error with pip not being able to install this python dependency.

**Update: the problem seems to be independent from the chosen python runtime, fails on python 3.8.x and 3.9.x

Step #5 - "Build and push final app image": [builder] Building wheels for collected packages: psycopg2
Step #5 - "Build and push final app image": [builder]   Building wheel for psycopg2 (setup.py): started
Step #5 - "Build and push final app image": [builder]   Building wheel for psycopg2 (setup.py): finished with status 'error'
Step #5 - "Build and push final app image": [builder]   error: subprocess-exited-with-error
Step #5 - "Build and push final app image": [builder]   
Step #5 - "Build and push final app image": [builder]   × python setup.py bdist_wheel did not run successfully.
Step #5 - "Build and push final app image": [builder]   │ exit code: 1
Step #5 - "Build and push final app image": [builder]   ╰─> [40 lines of output]
Step #5 - "Build and push final app image": [builder]       /layers/google.python.runtime/python/lib/python3.8/site-packages/setuptools/config/setupcfg.py:508: SetuptoolsDeprecationWarning: The license_file parameter is deprecated, use license_files instead.
Step #5 - "Build and push final app image": [builder]         warnings.warn(msg, warning_class)
Step #5 - "Build and push final app image": [builder]       running bdist_wheel
Step #5 - "Build and push final app image": [builder]       running build
Step #5 - "Build and push final app image": [builder]       running build_py
Step #5 - "Build and push final app image": [builder]       creating build
Step #5 - "Build and push final app image": [builder]       creating build/lib.linux-x86_64-cpython-38
Step #5 - "Build and push final app image": [builder]       creating build/lib.linux-x86_64-cpython-38/psycopg2
Step #5 - "Build and push final app image": [builder]       copying lib/tz.py -> build/lib.linux-x86_64-cpython-38/psycopg2
Step #5 - "Build and push final app image": [builder]       copying lib/sql.py -> build/lib.linux-x86_64-cpython-38/psycopg2
Step #5 - "Build and push final app image": [builder]       copying lib/errors.py -> build/lib.linux-x86_64-cpython-38/psycopg2
Step #5 - "Build and push final app image": [builder]       copying lib/extensions.py -> build/lib.linux-x86_64-cpython-38/psycopg2
Step #5 - "Build and push final app image": [builder]       copying lib/pool.py -> build/lib.linux-x86_64-cpython-38/psycopg2
Step #5 - "Build and push final app image": [builder]       copying lib/extras.py -> build/lib.linux-x86_64-cpython-38/psycopg2
Step #5 - "Build and push final app image": [builder]       copying lib/_ipaddress.py -> build/lib.linux-x86_64-cpython-38/psycopg2
Step #5 - "Build and push final app image": [builder]       copying lib/__init__.py -> build/lib.linux-x86_64-cpython-38/psycopg2
Step #5 - "Build and push final app image": [builder]       copying lib/_range.py -> build/lib.linux-x86_64-cpython-38/psycopg2
Step #5 - "Build and push final app image": [builder]       copying lib/errorcodes.py -> build/lib.linux-x86_64-cpython-38/psycopg2
Step #5 - "Build and push final app image": [builder]       copying lib/_json.py -> build/lib.linux-x86_64-cpython-38/psycopg2
Step #5 - "Build and push final app image": [builder]       running build_ext
Step #5 - "Build and push final app image": [builder]       building 'psycopg2._psycopg' extension
Step #5 - "Build and push final app image": [builder]       creating build/temp.linux-x86_64-cpython-38
Step #5 - "Build and push final app image": [builder]       creating build/temp.linux-x86_64-cpython-38/psycopg
Step #5 - "Build and push final app image": [builder]       x86_64-linux-gnu-gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fstack-protector-strong -g -Wformat -Werror=format-security -fPIC "-DPSYCOPG_VERSION=2.9.3 (dt dec pq3 ext lo64)" -DPSYCOPG_DEBUG=1 -DPG_VERSION_NUM=100022 -DHAVE_LO64=1 -DPSYCOPG_DEBUG=1 -I/opt/python3.8/include/python3.8 -I. -I/usr/include/postgresql -I/usr/include/postgresql/10/server -I/usr/include/libxml2 -I/usr/include/mit-krb5 -c psycopg/adapter_asis.c -o build/temp.linux-x86_64-cpython-38/psycopg/adapter_asis.o -Wdeclaration-after-statement
Step #5 - "Build and push final app image": [builder]       In file included from psycopg/adapter_asis.c:28:0:
Step #5 - "Build and push final app image": [builder]       ./psycopg/psycopg.h:35:10: fatal error: Python.h: No such file or directory
Step #5 - "Build and push final app image": [builder]        #include <Python.h>
Step #5 - "Build and push final app image": [builder]                 ^~~~~~~~~~
Step #5 - "Build and push final app image": [builder]       compilation terminated.
Step #5 - "Build and push final app image": [builder]       
Step #5 - "Build and push final app image": [builder]       It appears you are missing some prerequisite to build the package from source.
Step #5 - "Build and push final app image": [builder]       
Step #5 - "Build and push final app image": [builder]       You may install a binary package by installing 'psycopg2-binary' from PyPI.
Step #5 - "Build and push final app image": [builder]       If you want to install psycopg2 from source, please install the packages
Step #5 - "Build and push final app image": [builder]       required for the build and try again.
Step #5 - "Build and push final app image": [builder]       
Step #5 - "Build and push final app image": [builder]       For further information please check the 'doc/src/install.rst' file (also at
Step #5 - "Build and push final app image": [builder]       <https://www.psycopg.org/docs/install.html>).
Step #5 - "Build and push final app image": [builder]       
Step #5 - "Build and push final app image": [builder]       error: command '/usr/bin/x86_64-linux-gnu-gcc' failed with exit code 1
Step #5 - "Build and push final app image": [builder]       [end of output]
Step #5 - "Build and push final app image": [builder]   
Step #5 - "Build and push final app image": [builder]   note: This error originates from a subprocess, and is likely not a problem with pip.

[jama22]

Having trouble difficulty reproducing this locally with Python. Here's what i tried:

1. cloned https://github.com/GoogleCloudPlatform/buildpack-samples/tree/master/sample-python
2. Modified requirements.txt to add hiredis==2.0.0 as a requirement
3. tried to build the project locally using the pack cli:

pack build --builder=gcr.io/buildpacks/builder sample-python

and I was able to build the image locally. @rajjagesar you might be right, there could be some changes to the base OS layer but we might need some more information from ya'll to help with debugging.

A few things come to mind:

1. can ya'll try building your project locally using the pack CLI? This is what is used behind the scenes when you deploy with Cloud Run. You can grab the CLI from the CNB site here and you can use a similar command to build the project locally pack build --builder=gcr.io/builder=gcr.io/buildpacks/builder <project-name>

2. In your build logs, can you identify what the build and run images are? For example, in my logs I had: 295: Status: Downloaded newer image for gcr.io/buildpacks/builder:latest and 335: Status: Downloaded newer image for gcr.io/buildpacks/gcp/run:v1 in my output when i tried building it with gcloud builds submit --pack image=gcr.io/<project-id>/<image name>

3. What region are you deploying to

[jinglundong]

Our internal Python acceptance test is encountering the same issue. I won't be surprised if this is a bug in the buildpacks code (exposed to the GCP builder and used by Cloud Run).

We released gcr.io/buildpacks/builder@sha256:203e1131c6afcec446790ed1cbfd236c9966b8bdd474ab854653a5813d2e5a85 three days ago. The timeline seems to be aligned. You might be able to mitigate this by using a previous builder like: pack build --builder=gcr.io/buildpacks/builder@sha256:d58acb7bb8e48d36833170b301ada61e94293744df31eb5b225222b8c83d69ca. More previous builders can be found at gcr.io/buildpacks/builder.

It's pretty late here, and I don't feel the urgency of rollback the builder without knowing the exact impact. I will investigate and potentially roll it back tomorrow.

[rajjagesar]

@jama22:

1. Gonna get back to you on this.
2. Yes I can confirm that the pack command correctly references my "custom builder" image that I have created in a build step before using pack. This custom builder is needed to install OS level dependencies for Postgres: libpq-dev and python3-dev.
3. Europe

@jinglundong:

Unfortunately the proposed mitigation strategy of using an older builder image does not work. I've changed the reference in my "custom builder" dockerfile to different versions of the builder going as far back as May 2022. The same error keeps popping up.

[jama22]

Thanks for the followup @rajjagesar. Sorry I completely missed the fact that you're using a custom builder for your project.

1. Could you tell me more about your custom builder? Are you extending it off one of our builders published https://github.com/GoogleCloudPlatform/buildpacks/tree/main/builders ?
2. I also forgot to ask what your workflow is here. Are you deploying to Cloud Run directly with source deploy?

[jinglundong]

pack build emits "Successfully installed pip-22.2.2 setuptools-65.3.0 wheel-0.37.1". I verified that those are the latest versions.

Not sure why we are seeing two similar SetuptoolsDeprecationWarning in setuptools. setup.py shouldn't be used, but I see it being called (as the Setuptools warning claimed). According to the build log, we are calling python3 -m pip install not deprecated methods.

I also tried previous python builder version, and python runtime version. All of them hit the same error. It could be an issue in the logic of building the python interpreter (as we called language builder). I tried a pretty old python version --env GOOGLE_RUNTIME_VERSION=3.7.2, but didn't help. To pursue this route further, we could look at the last update timestamp of it. Maybe we updated them in place lately?

[jinglundong]

https://github.com/GoogleCloudPlatform/buildpacks/commit/d31a57a785671fead12bebd7772fdcffae5539f5 may have this fixed. It seems like an issue in setuptools v65 on python 3.10. Please let us know if this temporary fix works for you. Our internal acceptance tests have turned green.

[jama22]

Closing the issue since our internal tests have turned green