Pip causes problems when installing package(s) in docker image if kaniko cache is activated.

[christianbeland]

Pip causes problems when installing package(s) in docker image if kaniko cache is activated.

Actual behavior I want to build following image:

ARG docker_registry
FROM ${docker_registry}tensorflow/tensorflow:1.13.2-gpu-py3

USER root
RUN mkdir -p ~/.pip
ADD deployment/pip.conf ~/.pip/pip.conf

RUN pip3 install --upgrade pip

ADD requirements.txt requirements.txt
RUN echo "pip3 install requirements.txt"
RUN pip3 install -r requirements.txt

RUN mkdir /dist /app
ADD dist/${PACKAGE_NAME} /dist/
RUN echo "pip3 install /dist/*.whl"
RUN pip3 install /dist/*.whl --target /app

ENV PYTHONPATH "${PYTHONPATH}:/app"

Building using:

   /kaniko/executor --context $CI_PROJECT_DIR \
             --dockerfile $BASE_PATH/deployment/Dockerfile \
             --destination ${DOCKER_REGISTRY}/$FULL_IMAGE_NAME \
             --build-arg docker_registry="$DOCKER_REGISTRY_MIRROR/" \
             --build-arg http_proxy=$http_proxy \
             --build-arg https_proxy=$https_proxy \
             --build-arg no_proxy=$no_proxy \
             --insecure \
             --skip-tls-verify \
             --insecure-registry=${DOCKER_REGISTRY} \
             --cache=true \
             --cache-dir=$BASE_PATH/cache/

Causes:

INFO[0093] RUN echo "pip3 install requirements.txt"     
INFO[0093] cmd: /bin/bash                               
INFO[0093] args: [-c echo "pip3 install requirements.txt"] 
pip3 install requirements.txt
INFO[0093] Taking snapshot of full filesystem...        
INFO[0093] Resolving paths                              
INFO[0097] Pushing layer <repo>/<image_name>/cache:aa2b060b29af65575a6c64c9df27cd498e894526042ef0e700751a7df042ed29 to cache now 
INFO[0097] RUN pip3 install -r requirements.txt         
INFO[0097] cmd: /bin/bash                               
INFO[0097] args: [-c pip3 install -r requirements.txt]  
Collecting click
  Downloading click-7.1.1-py2.py3-none-any.whl (82 kB)
Collecting tokenization
  Downloading tokenization-1.0.7-py3-none-any.whl (10 kB)
Collecting cbor>=0.1.4
  Downloading cbor-1.0.0.tar.gz (20 kB)
Collecting typing
  Downloading typing-3.7.4.1-py3-none-any.whl (25 kB)
Collecting regex
  Downloading regex-2020.2.20-cp36-cp36m-manylinux2010_x86_64.whl (690 kB)
Building wheels for collected packages: cbor
  Building wheel for cbor (setup.py): started
  Building wheel for cbor (setup.py): finished with status 'done'
  Created wheel for cbor: filename=cbor-1.0.0-cp36-cp36m-linux_x86_64.whl size=51512 sha256=8d6ce77b82f6b6fc5d3e83f0eaa1b89f03357a0539430ebd033ee692b5624dd7
  Stored in directory: /root/.cache/pip/wheels/14/62/c6/8c93de62510c9c7a9c6d5b3a09fd74b4c3943ec0e32c431753
Successfully built cbor
Installing collected packages: click, regex, tokenization, cbor, typing
ERROR: Error checking for conflicts.
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/pip/_vendor/pkg_resources/__init__.py", line 3021, in _dep_map
    return self.__dep_map
  File "/usr/local/lib/python3.6/dist-packages/pip/_vendor/pkg_resources/__init__.py", line 2815, in __getattr__
    raise AttributeError(attr)
AttributeError: _DistInfoDistribution__dep_map

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/pip/_vendor/pkg_resources/__init__.py", line 3012, in _parsed_pkg_info
    return self._pkg_info
  File "/usr/local/lib/python3.6/dist-packages/pip/_vendor/pkg_resources/__init__.py", line 2815, in __getattr__
    raise AttributeError(attr)
AttributeError: _pkg_info

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/pip/_internal/commands/install.py", line 517, in _warn_about_conflicts
    package_set, _dep_info = check_install_conflicts(to_install)
  File "/usr/local/lib/python3.6/dist-packages/pip/_internal/operations/check.py", line 114, in check_install_conflicts
    package_set, _ = create_package_set_from_installed()
  File "/usr/local/lib/python3.6/dist-packages/pip/_internal/operations/check.py", line 53, in create_package_set_from_installed
    package_set[name] = PackageDetails(dist.version, dist.requires())
  File "/usr/local/lib/python3.6/dist-packages/pip/_vendor/pkg_resources/__init__.py", line 2736, in requires
    dm = self._dep_map
  File "/usr/local/lib/python3.6/dist-packages/pip/_vendor/pkg_resources/__init__.py", line 3023, in _dep_map
    self.__dep_map = self._compute_dependencies()
  File "/usr/local/lib/python3.6/dist-packages/pip/_vendor/pkg_resources/__init__.py", line 3032, in _compute_dependencies
    for req in self._parsed_pkg_info.get_all('Requires-Dist') or []:
  File "/usr/local/lib/python3.6/dist-packages/pip/_vendor/pkg_resources/__init__.py", line 3014, in _parsed_pkg_info
    metadata = self.get_metadata(self.PKG_INFO)
  File "/usr/local/lib/python3.6/dist-packages/pip/_vendor/pkg_resources/__init__.py", line 1895, in get_metadata
    raise KeyError("No metadata except PKG-INFO is available")
KeyError: 'No metadata except PKG-INFO is available'
Successfully installed cbor-1.0.0 click-7.1.1 regex-2020.2.20 tokenization-1.0.7 typing-3.7.4.1
INFO[0102] Taking snapshot of full filesystem...        
INFO[0102] Resolving paths

The image gets pushed to the repository but I cannot pull it. I receive following error:

Failed to pull image "<repo>/<image_name>:0.1.dev65-g9165108": rpc error: code = Unknown desc = failed to register layer: Error processing tar file(exit status 1): failed to mknod("/usr/bin/pip3", S_IFCHR, 0): file exists

Expected behavior If I comment out --cache=true and --cache-dir= from the kaniko build command, everything works fine. The build also works without problems using docker.

[swist]

Hitting the same issue intermittently on most versions from 0.17.1 to 0.19.0

[hochreitercorpuls]

we are also experiencing this issue resulting in exactly the same behaviour in our pipelines.

Hitting the same issue intermittently on most versions from 0.17.1 to 0.19.0

are you saying that 0.16 is safe?

[tejal29]

@christianbeland i am able to reproduce this error at my end. Looking into it now,.

[tejal29]

@christianbeland , i tried your dockerfile on latest master. Looks like this is fixed now. Can you please try,

gcr.io/kaniko-project/executor:edge
gcr.io/kaniko-project/executor:debug-edge

The dockerfile i used is

FROM tensorflow/tensorflow:1.13.2-gpu-py3
  
USER root
RUN mkdir -p ~/.pip

RUN pip3 install --upgrade pip

ADD requirements.txt requirements.txt
RUN echo "pip3 install requirements.txt"
RUN pip3 install -r requirements.txt

RUN mkdir /dist /app
ADD dist/*.whl /dist/
RUN echo "pip3 install /dist/*.whl"
RUN pip3 install /dist/*.whl --target /app

ENV PYTHONPATH "${PYTHONPATH}:/app"

[christianbeland]

@tejal29 I just tried and still have the issue.

You may have to try the build twice. I noticed the first build often work.

[tejal29]

okay let me do that. Thanks for confirming so fast

[christianbeland]

You should probably remove the 'fixed-needs-verfication' tag.

[astleychen]

I ran into same issue here and confirmed it's fixed on gcr.io/kaniko-project/executor:debug-edge.

[christianbeland]

I tested gcr.io/kaniko-project/executor:debug-edge again this morning and still does not work.

I get:

in _compute_dependencies for req in self._parsed_pkg_info.get_all('Requires-Dist') or []: File "/opt/conda/lib/python3.6/site-packages/pip/_vendor/pkg_resources/__init__.py", line 3014, in _parsed_pkg_info metadata = self.get_metadata(self.PKG_INFO) File "/opt/conda/lib/python3.6/site-packages/pip/_vendor/pkg_resources/__init__.py", line 1895, in get_metadata raise KeyError("No metadata except PKG-INFO is available") KeyError: 'No metadata except PKG-INFO is available'

[figroc]

Has this been fixed or not? We currently use 1.6.0 but encounter the same issue.

[TheMatrix97]

Facing a similar issue here. In my case if cache=true it throws the following error when I try to run python -m pip install kubernetes==11.0.0 with python3.11

Processing triggers for libc-bin (2.36-9+deb12u4) ...
error: externally-managed-environment
× This environment is externally managed
╰─> To install Python packages system-wide, try apt install
    python3-xyz, where xyz is the package you are trying to
    install.
    
    If you wish to install a non-Debian-packaged Python package,
    create a virtual environment using python3 -m venv path/to/venv.
    Then use path/to/venv/bin/python and path/to/venv/bin/pip. Make
    sure you have python3-full installed.
    
    If you wish to install a non-Debian packaged Python application,
    it may be easiest to use pipx install xyz, which will manage a
    virtual environment for you. Make sure you have pipx installed.
    
    See /usr/share/doc/python3.11/README.venv for more information.
note: If you believe this is a mistake, please contact your Python installation or OS distribution provider. You can override this, at the risk of breaking your Python installation or OS, by passing --break-system-packages.
hint: See PEP 668 for the detailed specification.
error building image: error building stage: failed to execute command: waiting for process to exit: exit status 1

The docker image is based in ubuntu:22.04

If I set the attribute cache=false the image is built without any issue... Just as a note, I'm using executor v1.14.0-debug if I try to use the latest one v1.21.1-debug it fails for both cache = false and true....

Any idea? Thanks!

[TheMatrix97]

Facing a similar issue here. In my case if cache=true it throws the following error when I try to run python -m pip install kubernetes==11.0.0 with python3.11

Processing triggers for libc-bin (2.36-9+deb12u4) ...
error: externally-managed-environment
× This environment is externally managed
╰─> To install Python packages system-wide, try apt install
    python3-xyz, where xyz is the package you are trying to
    install.
    
    If you wish to install a non-Debian-packaged Python package,
    create a virtual environment using python3 -m venv path/to/venv.
    Then use path/to/venv/bin/python and path/to/venv/bin/pip. Make
    sure you have python3-full installed.
    
    If you wish to install a non-Debian packaged Python application,
    it may be easiest to use pipx install xyz, which will manage a
    virtual environment for you. Make sure you have pipx installed.
    
    See /usr/share/doc/python3.11/README.venv for more information.
note: If you believe this is a mistake, please contact your Python installation or OS distribution provider. You can override this, at the risk of breaking your Python installation or OS, by passing --break-system-packages.
hint: See PEP 668 for the detailed specification.
error building image: error building stage: failed to execute command: waiting for process to exit: exit status 1

The docker image is based in ubuntu:22.04

If I set the attribute cache=false the image is built without any issue... Just as a note, I'm using executor v1.14.0-debug if I try to use the latest one v1.21.1-debug it fails for both cache = false and true....

Any idea? Thanks!

Well... I just solved it, I'm running different builds on the same container for different Dockerfile, and I was missing the --cleanup flag, adding this fixes the issue

[ch9hn]

Hello, we are facing exact the same issue - we can't install prebuild Python Wheels with Kaniko. With Docker (BuildKit) its working fine. We also try to disable the cache and run it also with --cleanup, but had no luck.

Our Kaniko Version is 1.21.0-debug, execution via Gitlab CI/CD.

INFO[0579] Taking snapshot of full filesystem...        
INFO[0735] RUN --mount=type=bind,from=flash-attn-builder,src=/usr/src/flash-attention-v2,target=/usr/src/flash-attention-v2     --mount=type=cache,target=/root/.cache/pip     pip install /usr/src/flash-attention-v2/*.whl --no-cache-dir     && python3 -m pip cache purge      && rm -rf /tmp/* RUN ldconfig /usr/local/cuda-11.8/compat/ 
INFO[0735] Cmd: /bin/sh                                 
INFO[0735] Args: [-c pip install /usr/src/flash-attention-v2/*.whl --no-cache-dir     && python3 -m pip cache purge      && rm -rf /tmp/* RUN ldconfig /usr/local/cuda-11.8/compat/] 
INFO[0735] Running: [/bin/sh -c pip install /usr/src/flash-attention-v2/*.whl --no-cache-dir     && python3 -m pip cache purge      && rm -rf /tmp/* RUN ldconfig /usr/local/cuda-11.8/compat/] 
WARNING: Requirement '/usr/src/flash-attention-v2/*.whl' looks like a filename, but the file does not exist
ERROR: *.whl is not a valid wheel filename.
error building image: error building stage: failed to execute command: waiting for process to exit: exit status 1

[ch9hn]

Found the source of that issue: https://github.com/GoogleContainerTools/kaniko/issues/1568

The unsupported options are not failing the build, the files are not mounted and therefore Python can't install the wheel.