pip icon indicating copy to clipboard operation
pip copied to clipboard

Published 20 hours ago • verified publisher icon pypa

pip should not cache large files in /tmp or TMPDIR

Open lee-b opened this issue 7 months ago • 14 comments

Description

Pip violates system specifications, and therefore essentially only works by accident right now, when it "gets lucky" with packages being small enough for /tmp.

This becomes a serious problem when working with larger pip installations, such as for vllm, pytorch with cuda acceleration, etc.

#5816 was closed with the advice to ensure that /tmp is large enough. #4462 was on the same topic and also closed without action. Pip also blames the user/system, with this output:

OSError: [Errno 28] No space left on device
...
note: This error originates from a subprocess, and is likely not a problem with pip.

However, these analyses are NOT correct.

If one reads the Linux file-hierarchy (7) man page specification (i.e., runs man 7 file-hierarchy) on Linux (Debian 12, at least), it states:

       /tmp/
           The place for small temporary files. This directory is usually mounted as a "tmpfs" instance, and should
           hence not be used for larger files. (Use /var/tmp/ for larger files.)

This document also refers readers to:

https://systemd.io/TEMPORARY_DIRECTORIES/

Which similarly states:

/tmp/ and /var/tmp/ are two world-writable directories Linux systems provide for temporary files. The former is typically on tmpfs and thus backed by RAM/swap, and flushed out on each reboot. The latter is typically a proper, persistent file system, and thus backed by physical storage. This means:

    /tmp/ should be used for smaller, size-bounded files only; /var/tmp/ should be used for everything else.

Moreover, the data in question seems to be cached data, not normal temporary files, and should therefore go in /var/cache (probably only if daemon is writing it, I believe), or into the XDG cache directory (e.g., ~/.cache/pip/).

Expected behavior

Pip should follow all relevant specifications when creating files, rather than putting large files in the wrong place and overloading filesystems that are not intended for large files.

pip version

24.0

Python version

3.12.2

OS

Debian 12

How to Reproduce

bin/pip3 install vllm when /tmp has 1.7GB available.

Output

Collecting vllm
  Using cached vllm-0.5.2-cp38-abi3-manylinux1_x86_64.whl.metadata (1.8 kB)
Collecting aiohttp (from vllm)
  Downloading aiohttp-3.9.5-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (7.5 kB)
Collecting cmake>=3.21 (from vllm)
  Using cached cmake-3.30.0-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (6.1 kB)
Collecting fastapi (from vllm)
  Using cached fastapi-0.111.1-py3-none-any.whl.metadata (26 kB)
Collecting filelock>=3.10.4 (from vllm)
  Using cached filelock-3.15.4-py3-none-any.whl.metadata (2.9 kB)
Collecting lm-format-enforcer==0.10.3 (from vllm)
  Using cached lm_format_enforcer-0.10.3-py3-none-any.whl.metadata (16 kB)
Collecting ninja (from vllm)
  Using cached ninja-1.11.1.1-py2.py3-none-manylinux1_x86_64.manylinux_2_5_x86_64.whl.metadata (5.3 kB)
Collecting numpy<2.0.0 (from vllm)
  Using cached numpy-1.26.4-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (61 kB)
Collecting nvidia-ml-py (from vllm)
  Using cached nvidia_ml_py-12.555.43-py3-none-any.whl.metadata (8.6 kB)
Collecting openai (from vllm)
  Using cached openai-1.36.1-py3-none-any.whl.metadata (22 kB)
Collecting outlines<0.1,>=0.0.43 (from vllm)
  Using cached outlines-0.0.46-py3-none-any.whl.metadata (15 kB)
Collecting pillow (from vllm)
  Using cached pillow-10.4.0-cp312-cp312-manylinux_2_28_x86_64.whl.metadata (9.2 kB)
Collecting prometheus-fastapi-instrumentator>=7.0.0 (from vllm)
  Using cached prometheus_fastapi_instrumentator-7.0.0-py3-none-any.whl.metadata (13 kB)
Collecting prometheus-client>=0.18.0 (from vllm)
  Using cached prometheus_client-0.20.0-py3-none-any.whl.metadata (1.8 kB)
Collecting psutil (from vllm)
  Using cached psutil-6.0.0-cp36-abi3-manylinux_2_12_x86_64.manylinux2010_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (21 kB)
Collecting py-cpuinfo (from vllm)
  Using cached py_cpuinfo-9.0.0-py3-none-any.whl.metadata (794 bytes)
Collecting pydantic>=2.0 (from vllm)
  Using cached pydantic-2.8.2-py3-none-any.whl.metadata (125 kB)
Collecting pyzmq (from vllm)
  Downloading pyzmq-26.0.3-cp312-cp312-manylinux_2_28_x86_64.whl.metadata (6.1 kB)
Collecting ray>=2.9 (from vllm)
  Downloading ray-2.32.0-cp312-cp312-manylinux2014_x86_64.whl.metadata (13 kB)
Collecting requests (from vllm)
  Using cached requests-2.32.3-py3-none-any.whl.metadata (4.6 kB)
Collecting sentencepiece (from vllm)
  Downloading sentencepiece-0.2.0-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (7.7 kB)
Collecting tiktoken>=0.6.0 (from vllm)
  Downloading tiktoken-0.7.0-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (6.6 kB)
Collecting tokenizers>=0.19.1 (from vllm)
  Downloading tokenizers-0.19.1-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (6.7 kB)
Collecting torch==2.3.1 (from vllm)
  Downloading torch-2.3.1-cp312-cp312-manylinux1_x86_64.whl.metadata (26 kB)
Collecting torchvision==0.18.1 (from vllm)
  Downloading torchvision-0.18.1-cp312-cp312-manylinux1_x86_64.whl.metadata (6.6 kB)
Collecting tqdm (from vllm)
  Using cached tqdm-4.66.4-py3-none-any.whl.metadata (57 kB)
Collecting transformers>=4.42.4 (from vllm)
  Using cached transformers-4.42.4-py3-none-any.whl.metadata (43 kB)
Collecting typing-extensions (from vllm)
  Using cached typing_extensions-4.12.2-py3-none-any.whl.metadata (3.0 kB)
Collecting uvicorn[standard] (from vllm)
  Using cached uvicorn-0.30.3-py3-none-any.whl.metadata (6.5 kB)
INFO: pip is looking at multiple versions of vllm to determine which version is compatible with other requirements. This could take a while.
Collecting vllm
  Downloading vllm-0.5.1.tar.gz (790 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 790.6/790.6 kB 7.0 MB/s eta 0:00:00
  Installing build dependencies ... error
  error: subprocess-exited-with-error

  × pip subprocess to install build dependencies did not run successfully.
  │ exit code: 1
  ╰─> [64 lines of output]
      Collecting cmake>=3.21
        Using cached cmake-3.30.0-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (6.1 kB)
      Collecting ninja
        Using cached ninja-1.11.1.1-py2.py3-none-manylinux1_x86_64.manylinux_2_5_x86_64.whl.metadata (5.3 kB)
      Collecting packaging
        Using cached packaging-24.1-py3-none-any.whl.metadata (3.2 kB)
      Collecting setuptools>=49.4.0
        Downloading setuptools-71.0.4-py3-none-any.whl.metadata (6.5 kB)
      Collecting torch==2.3.0
        Downloading torch-2.3.0-cp312-cp312-manylinux1_x86_64.whl.metadata (26 kB)
      Collecting wheel
        Using cached wheel-0.43.0-py3-none-any.whl.metadata (2.2 kB)
      Collecting filelock (from torch==2.3.0)
        Using cached filelock-3.15.4-py3-none-any.whl.metadata (2.9 kB)
      Collecting typing-extensions>=4.8.0 (from torch==2.3.0)
        Using cached typing_extensions-4.12.2-py3-none-any.whl.metadata (3.0 kB)
      Collecting sympy (from torch==2.3.0)
        Using cached sympy-1.13.1-py3-none-any.whl.metadata (12 kB)
      Collecting networkx (from torch==2.3.0)
        Using cached networkx-3.3-py3-none-any.whl.metadata (5.1 kB)
      Collecting jinja2 (from torch==2.3.0)
        Using cached jinja2-3.1.4-py3-none-any.whl.metadata (2.6 kB)
      Collecting fsspec (from torch==2.3.0)
        Using cached fsspec-2024.6.1-py3-none-any.whl.metadata (11 kB)
      Collecting nvidia-cuda-nvrtc-cu12==12.1.105 (from torch==2.3.0)
        Using cached nvidia_cuda_nvrtc_cu12-12.1.105-py3-none-manylinux1_x86_64.whl.metadata (1.5 kB)
      Collecting nvidia-cuda-runtime-cu12==12.1.105 (from torch==2.3.0)
        Using cached nvidia_cuda_runtime_cu12-12.1.105-py3-none-manylinux1_x86_64.whl.metadata (1.5 kB)
      Collecting nvidia-cuda-cupti-cu12==12.1.105 (from torch==2.3.0)
        Using cached nvidia_cuda_cupti_cu12-12.1.105-py3-none-manylinux1_x86_64.whl.metadata (1.6 kB)
      Collecting nvidia-cudnn-cu12==8.9.2.26 (from torch==2.3.0)
        Using cached nvidia_cudnn_cu12-8.9.2.26-py3-none-manylinux1_x86_64.whl.metadata (1.6 kB)
      Collecting nvidia-cublas-cu12==12.1.3.1 (from torch==2.3.0)
        Using cached nvidia_cublas_cu12-12.1.3.1-py3-none-manylinux1_x86_64.whl.metadata (1.5 kB)
      Collecting nvidia-cufft-cu12==11.0.2.54 (from torch==2.3.0)
        Using cached nvidia_cufft_cu12-11.0.2.54-py3-none-manylinux1_x86_64.whl.metadata (1.5 kB)
      Collecting nvidia-curand-cu12==10.3.2.106 (from torch==2.3.0)
        Using cached nvidia_curand_cu12-10.3.2.106-py3-none-manylinux1_x86_64.whl.metadata (1.5 kB)
      Collecting nvidia-cusolver-cu12==11.4.5.107 (from torch==2.3.0)
        Using cached nvidia_cusolver_cu12-11.4.5.107-py3-none-manylinux1_x86_64.whl.metadata (1.6 kB)
      Collecting nvidia-cusparse-cu12==12.1.0.106 (from torch==2.3.0)
        Using cached nvidia_cusparse_cu12-12.1.0.106-py3-none-manylinux1_x86_64.whl.metadata (1.6 kB)
      Collecting nvidia-nccl-cu12==2.20.5 (from torch==2.3.0)
        Using cached nvidia_nccl_cu12-2.20.5-py3-none-manylinux2014_x86_64.whl.metadata (1.8 kB)
      Collecting nvidia-nvtx-cu12==12.1.105 (from torch==2.3.0)
        Using cached nvidia_nvtx_cu12-12.1.105-py3-none-manylinux1_x86_64.whl.metadata (1.7 kB)
      Collecting nvidia-nvjitlink-cu12 (from nvidia-cusolver-cu12==11.4.5.107->torch==2.3.0)
        Using cached nvidia_nvjitlink_cu12-12.5.82-py3-none-manylinux2014_x86_64.whl.metadata (1.5 kB)
      Collecting MarkupSafe>=2.0 (from jinja2->torch==2.3.0)
        Using cached MarkupSafe-2.1.5-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (3.0 kB)
      Collecting mpmath<1.4,>=1.1.0 (from sympy->torch==2.3.0)
        Using cached mpmath-1.3.0-py3-none-any.whl.metadata (8.6 kB)
      Downloading torch-2.3.0-cp312-cp312-manylinux1_x86_64.whl (779.1 MB)
         ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 779.1/779.1 MB 5.3 MB/s eta 0:00:00
      Using cached nvidia_cublas_cu12-12.1.3.1-py3-none-manylinux1_x86_64.whl (410.6 MB)
      Using cached nvidia_cuda_cupti_cu12-12.1.105-py3-none-manylinux1_x86_64.whl (14.1 MB)
      Using cached nvidia_cuda_nvrtc_cu12-12.1.105-py3-none-manylinux1_x86_64.whl (23.7 MB)
      Using cached nvidia_cuda_runtime_cu12-12.1.105-py3-none-manylinux1_x86_64.whl (823 kB)
      Using cached nvidia_cudnn_cu12-8.9.2.26-py3-none-manylinux1_x86_64.whl (731.7 MB)
      ERROR: Could not install packages due to an OSError: [Errno 28] No space left on device


      [notice] A new release of pip is available: 24.0 -> 24.1.2
      [notice] To update, run: /mnt/nvme1/home/lb/.local/.venvs/vllm/bin/python3 -m pip install --upgrade pip
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
error: subprocess-exited-with-error

Code of Conduct

lee-b avatar Jul 21 '24 10:07 lee-b