runtime icon indicating copy to clipboard operation
runtime copied to clipboard
verified publisher icon dotnet

Failed to install runtime_python_requirements

Open carlossanlop opened this issue 8 months ago • 8 comments

The runtime-community CI legs in the 8.0 branch are failing with two errors:

chown: changing ownership of '/home/helixbot/.vsts-env/*': Operation not permitted

and

Failed to install runtime_python_requirements

The second one is what's being captured by the CI system, but seems to be a consequence of the first one.

Build Information

Build: https://dev.azure.com/dnceng-public/public/_build/results?buildId=1017225 Build error leg or test failing: linux-armv6 Release AllSubsets_Mono

Error Message

{
  "ErrorMessage" : "Failed to install runtime_python_requirements",
  "BuildRetry" : false,
  "ExcludeConsoleLog" : false
}
  • PR: https://github.com/dotnet/runtime/pull/114668
  • Queue: linux-armv6 Release AllSubsets_Mono
  • Job result: https://dev.azure.com/dnceng-public/public/_build/results?buildId=1017225&view=logs&j=db8e93a1-262e-5037-1aaf-a14621311ed2&t=c2d6c85a-08d6-524c-58e0-6a79597938c0&l=91
  • Log file: https://helixr18s23ayyeko0k025g8.blob.core.windows.net/dotnet-runtime-refs-heads-release-80-3869953c7bbf4c6f9a/Common.Tests/1/console.c97d120f.log?helixlogtype=result
  • Output:
+ chown -R helixbot /home/helixbot/.vsts-env
chown: changing ownership of '/home/helixbot/.vsts-env/include/python3.7m': Operation not permitted
chown: changing ownership of '/home/helixbot/.vsts-env/include': Operation not permitted
chown: changing ownership of '/home/helixbot/.vsts-env/lib/python3.7/orig-prefix.txt': Operation not permitted
chown: changing ownership of '/home/helixbot/.vsts-env/lib/python3.7/collections': Operation not permitted
chown: changing ownership of '/home/helixbot/.vsts-env/lib/python3.7/site-packages/wheel-0.40.0.dist-info/INSTALLER': Operation not permitted
chown: changing ownership of '/home/helixbot/.vsts-env/lib/python3.7/site-packages/wheel-0.40.0.dist-info/entry_points.txt': Operation not permitted
chown: changing ownership of '/home/helixbot/.vsts-env/lib/python3.7/site-packages/wheel-0.40.0.dist-info/METADATA': Operation not permitted
...
...
...
  Preparing metadata (pyproject.toml): started
  Preparing metadata (pyproject.toml): finished with status 'error'
  error: subprocess-exited-with-error
  
  × Preparing metadata (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> [6 lines of output]
      
      Cargo, the Rust package manager, is not installed or is not on PATH.
      This package requires Rust and Cargo to compile extensions. Install it through
      the system's package manager or via https://rustup.rs/
      
      Checking for Rust toolchain....
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
error: metadata-generation-failed

× Encountered error while generating package metadata.
╰─> See above for output.

note: This is an issue with the package mentioned above, not pip.
hint: See above for details.
Failed to install runtime_python_requirements
+ echo Failed to install runtime_python_requirements
+ exit 1

[END EXECUTION]
Exit Code:1

Known issue validation

Build: :mag_right: https://dev.azure.com/dnceng-public/public/_build/results?buildId=1017225 Error message validated: [Failed to install runtime_python_requirements] Result validation: :white_check_mark: Known issue matched with the provided build. Validation performed at: 4/22/2025 7:12:44 PM UTC

Report

Build Definition Test Pull Request
1221895 dotnet/runtime System.IO.Compression.ZipFile.Tests.WorkItemExecution
1221489 dotnet/runtime System.Reflection.InvokeEmit.Tests.WorkItemExecution
1220597 dotnet/runtime System.Formats.Tar.Manual.Tests.WorkItemExecution
1220518 dotnet/runtime System.IO.MemoryMappedFiles.Tests.WorkItemExecution
1213694 dotnet/runtime System.IO.Ports.Tests.WorkItemExecution
1213691 dotnet/runtime System.Net.Http.Enterprise.Tests.WorkItemExecution
1213647 dotnet/runtime System.Formats.Tar.Tests.WorkItemExecution
1213633 dotnet/runtime System.Globalization.Tests.WorkItemExecution
1213603 dotnet/runtime System.IO.FileSystem.Primitives.Tests.WorkItemExecution
1205994 dotnet/runtime System.Diagnostics.TextWriterTraceListener.Tests.WorkItemExecution
1203813 dotnet/runtime System.IO.FileSystem.Manual.Tests.WorkItemExecution
1198830 dotnet/runtime System.IO.FileSystem.Manual.Tests.WorkItemExecution

Summary

24-Hour Hit Count 7-Day Hit Count 1-Month Count
0 4 12

carlossanlop avatar Apr 22 '25 19:04 carlossanlop

Tagging subscribers to this area: @akoeplinger, @matouskozak See info in area-owners.md if you want to be subscribed.

dotnet-policy-service[bot] avatar Apr 23 '25 15:04 dotnet-policy-service[bot]

Lots of recent failures on this one, impacting many test runs, e.g.: https://github.com/dotnet/runtime/pull/121261/checks?check_run_id=54388775664

Building wheels for collected packages: psutil
  Building wheel for psutil (pyproject.toml): started
  Building wheel for psutil (pyproject.toml): finished with status 'error'
  error: subprocess-exited-with-error
  
  × Building wheel for psutil (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> [54 lines of output]
      /tmp/pip-build-env-uwcyjtbp/overlay/lib/python3.12/site-packages/setuptools/dist.py:759: SetuptoolsDeprecationWarning: License classifiers are deprecated.
      !!
      
              ********************************************************************************
              Please consider removing the following classifiers in favor of a SPDX license expression:
      
              License :: OSI Approved :: BSD License
      
              See https://packaging.python.org/en/latest/guides/writing-pyproject-toml/#license for details.
              ********************************************************************************
      
      !!
        self._finalize_license_expression()

MihaZupan avatar Nov 03 '25 18:11 MihaZupan

Tagging subscribers to this area: @dotnet/runtime-infrastructure See info in area-owners.md if you want to be subscribed.

dotnet-policy-service[bot] avatar Nov 04 '25 13:11 dotnet-policy-service[bot]

I believe #121331 was the temporary workaround for this

jakobbotsch avatar Nov 04 '25 18:11 jakobbotsch

The real error is at the end psutil could not be installed from sources because gcc is not installed.:

(2.1.1)
Requirement already satisfied: PyJWT<3,>=1.0.0 in /home/helixbot/.vsts-env/lib/python3.12/site-packages (from PyJWT[crypto]<3,>=1.0.0->msal>=1.30.0->azure-identity>=1.16->-r /root/helix/scripts/runtime_python_requirements.txt (line 5)) (2.10.1)
Building wheels for collected packages: psutil
  Building wheel for psutil (pyproject.toml): started
  Building wheel for psutil (pyproject.toml): finished with status 'error'
  error: subprocess-exited-with-error
  
  × Building wheel for psutil (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> [54 lines of output]
      /tmp/pip-build-env-pp2vmpwe/overlay/lib/python3.12/site-packages/setuptools/dist.py:759: SetuptoolsDeprecationWarning: License classifiers are deprecated.
      !!
      
              ********************************************************************************
              Please consider removing the following classifiers in favor of a SPDX license expression:
      
              License :: OSI Approved :: BSD License
      
              See https://packaging.python.org/en/latest/guides/writing-pyproject-toml/#license for details.
              ********************************************************************************
      
      !!
        self._finalize_license_expression()
      running bdist_wheel
      running build
      running build_py
      creating build/lib.linux-x86_64-cpython-312/psutil
      copying psutil/_pswindows.py -> build/lib.linux-x86_64-cpython-312/psutil
      copying psutil/__init__.py -> build/lib.linux-x86_64-cpython-312/psutil
      copying psutil/_psaix.py -> build/lib.linux-x86_64-cpython-312/psutil
      copying psutil/_pssunos.py -> build/lib.linux-x86_64-cpython-312/psutil
      copying psutil/_common.py -> build/lib.linux-x86_64-cpython-312/psutil
      copying psutil/_psosx.py -> build/lib.linux-x86_64-cpython-312/psutil
      copying psutil/_psbsd.py -> build/lib.linux-x86_64-cpython-312/psutil
      copying psutil/_psposix.py -> build/lib.linux-x86_64-cpython-312/psutil
      copying psutil/_pslinux.py -> build/lib.linux-x86_64-cpython-312/psutil
      creating build/lib.linux-x86_64-cpython-312/psutil/tests
      copying psutil/tests/test_misc.py -> build/lib.linux-x86_64-cpython-312/psutil/tests
      copying psutil/tests/test_process.py -> build/lib.linux-x86_64-cpython-312/psutil/tests
      copying psutil/tests/test_osx.py -> build/lib.linux-x86_64-cpython-312/psutil/tests
      copying psutil/tests/test_contracts.py -> build/lib.linux-x86_64-cpython-312/psutil/tests
      copying psutil/tests/__init__.py -> build/lib.linux-x86_64-cpython-312/psutil/tests
      copying psutil/tests/test_posix.py -> build/lib.linux-x86_64-cpython-312/psutil/tests
      copying psutil/tests/__main__.py -> build/lib.linux-x86_64-cpython-312/psutil/tests
      copying psutil/tests/test_sunos.py -> build/lib.linux-x86_64-cpython-312/psutil/tests
      copying psutil/tests/test_linux.py -> build/lib.linux-x86_64-cpython-312/psutil/tests
      copying psutil/tests/test_process_all.py -> build/lib.linux-x86_64-cpython-312/psutil/tests
      copying psutil/tests/test_memleaks.py -> build/lib.linux-x86_64-cpython-312/psutil/tests
      copying psutil/tests/test_bsd.py -> build/lib.linux-x86_64-cpython-312/psutil/tests
      copying psutil/tests/test_connections.py -> build/lib.linux-x86_64-cpython-312/psutil/tests
      copying psutil/tests/test_scripts.py -> build/lib.linux-x86_64-cpython-312/psutil/tests
      copying psutil/tests/test_testutils.py -> build/lib.linux-x86_64-cpython-312/psutil/tests
      copying psutil/tests/test_aix.py -> build/lib.linux-x86_64-cpython-312/psutil/tests
      copying psutil/tests/test_windows.py -> build/lib.linux-x86_64-cpython-312/psutil/tests
      copying psutil/tests/test_unicode.py -> build/lib.linux-x86_64-cpython-312/psutil/tests
      copying psutil/tests/test_system.py -> build/lib.linux-x86_64-cpython-312/psutil/tests
      running build_ext
      building 'psutil._psutil_linux' extension
      creating build/temp.linux-x86_64-cpython-312/psutil
      creating build/temp.linux-x86_64-cpython-312/psutil/arch/linux
      cc -fno-strict-overflow -Wsign-compare -DNDEBUG -g -O3 -Wall -fPIC -DPSUTIL_POSIX=1 -DPSUTIL_SIZEOF_PID_T=4 -DPSUTIL_VERSION=700 -DPy_LIMITED_API=0x03060000 -DPSUTIL_ETHTOOL_MISSING_TYPES=1 -DPSUTIL_LINUX=1 -I/home/helixbot/.vsts-env/include -I/usr/include/python3.12 -c psutil/_psutil_common.c -o build/temp.linux-x86_64-cpython-312/psutil/_psutil_common.o
      psutil could not be installed from sources because gcc is not installed. Try running:
        sudo apk add gcc p3-dev musl-dev linux-headers
      error: command 'cc' failed: No such file or directory
      [end of output]

@akoeplinger, @chcosta, please add build-base and linux-headers to the list. Isolated repro:

$ docker build - -t foo <<EOF
FROM alpine:3.22

RUN apk add --no-cache \
    py3-pip \
    python3-dev \
    build-base \
    linux-headers

RUN python3 -m venv /venv
ENV PATH="/venv/bin:$PATH"

RUN pip install --no-cache-dir psutil
EOF

...
# succeeds:
 => exporting to image                                                                                                                                                                                                                          0.3s 
 => => exporting layers

am11 avatar Nov 04 '25 18:11 am11

@am11 is #121331 not resolving the problem? The real error is that psutil stopped including 32 bit prebuilts (https://github.com/giampaolo/psutil/blob/master/HISTORY.rst#712) which exposed an issue with the Helix images that was originally implemented because of lack of arm64 support in some python packages. When prebuilts are not found, the package manager tries to build them from source which surfaces as the error you listed. We temporarily pinned to an earlier psutil version which includes the prebuilts and that now necessitates pinning the alpine images so that the psutil version matches. The real fix for this will rollout this week (fingers crossed), and then #121331 will no longer be necessary.

chcosta avatar Nov 04 '25 19:11 chcosta

@chcosta, yes that downgrade was a workaround and fallback to source build is also known. Seeing that this issue is opened for about seven months, it caught my attention. The fix turned out to be literally one liner: apk add py3-pip python3-dev build-base linux-headers, which I've shown above.

am11 avatar Nov 04 '25 19:11 am11

@am11 , sure, i have no opinion on this long standing issue, more specifically, I assumed I was tagged in regards to psutil which does not need additional dependencies. With respect to the rest, I'll peace out :)

chcosta avatar Nov 05 '25 00:11 chcosta