pip icon indicating copy to clipboard operation
pip copied to clipboard
verified publisher icon pypa

OSError: [Errno 18] Invalid cross-device link and OSError: [Errno 22] Invalid argument:

Open tocosastalo opened this issue 6 years ago • 8 comments

Environment

  • pip version: 19.2.3
  • Python version: 3.6
  • OS: centos-release-7-5.1804.el7.centos.2.x86_64
  • Docker version: 18.09.7, build 2d0083d

Description

I would like to uninstall python package using pip during docker container build, but always I run into the same two errors: OSError: [Errno 18] Invalid cross-device link and OSError: [Errno 22] Invalid argument.

Expected behavior

Python library should be uninstalled without errors.

How to Reproduce

  1. I created demo dockerfile:
FROM python:3.6

WORKDIR /app

RUN pip install requests
RUN pip uninstall requests --yes
  1. Then I run: docker build -t demo .
  2. Everything is fine until the uninstall command (see output).

Dockerfile in other tested environment (Win10, docker 19.03.01, python 3.6.8, pip 18.1) works.

Output

[xjakubek@eqs test]$ docker build -t demo .
Sending build context to Docker daemon  73.03MB
Step 1/4 : FROM python:3.6
 ---> 1c515a624542
Step 2/4 : WORKDIR /app
 ---> Running in e74a4eb969be
Removing intermediate container e74a4eb969be
 ---> a47d7cc9bb48
Step 3/4 : RUN pip install requests
 ---> Running in a725f5935de3
Collecting requests
  Downloading https://files.pythonhosted.org/packages/51/bd/23c926cd341ea6b7dd0b2a00aba99ae0f828be89d72b2190f27c11d4b7fb/requests-2.22.0-py2.py3-none-any.whl (57kB)
Collecting idna<2.9,>=2.5 (from requests)
  Downloading https://files.pythonhosted.org/packages/14/2c/cd551d81dbe15200be1cf41cd03869a46fe7226e7450af7a6545bfc474c9/idna-2.8-py2.py3-none-any.whl (58kB)
Collecting certifi>=2017.4.17 (from requests)
  Downloading https://files.pythonhosted.org/packages/69/1b/b853c7a9d4f6a6d00749e94eb6f3a041e342a885b87340b79c1ef73e3a78/certifi-2019.6.16-py2.py3-none-any.whl (157kB)
Collecting chardet<3.1.0,>=3.0.2 (from requests)
  Downloading https://files.pythonhosted.org/packages/bc/a9/01ffebfb562e4274b6487b4bb1ddec7ca55ec7510b22e4c51f14098443b8/chardet-3.0.4-py2.py3-none-any.whl (133kB)
Collecting urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 (from requests)
  Downloading https://files.pythonhosted.org/packages/e6/60/247f23a7121ae632d62811ba7f273d0e58972d75e58a94d329d51550a47d/urllib3-1.25.3-py2.py3-none-any.whl (150kB)
Installing collected packages: idna, certifi, chardet, urllib3, requests
Successfully installed certifi-2019.6.16 chardet-3.0.4 idna-2.8 requests-2.22.0 urllib3-1.25.3
Removing intermediate container a725f5935de3
 ---> 2898dc3e30b5
Step 4/4 : RUN pip uninstall requests --yes
 ---> Running in 752ab918d7c2
Uninstalling requests-2.22.0:
ERROR: Exception:
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/shutil.py", line 550, in move
    os.rename(src, real_dst)
OSError: [Errno 18] Invalid cross-device link: '/usr/local/lib/python3.6/site-packages/requests-2.22.0.dist-info/' -> '/usr/local/lib/python3.6/site-packages/~equests-2.22.0.dist-info'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/pip/_internal/cli/base_command.py", line 188, in main
    status = self.run(options, args)
  File "/usr/local/lib/python3.6/site-packages/pip/_internal/commands/uninstall.py", line 75, in run
    auto_confirm=options.yes, verbose=self.verbosity > 0,
  File "/usr/local/lib/python3.6/site-packages/pip/_internal/req/req_install.py", line 831, in uninstall
    uninstalled_pathset.remove(auto_confirm, verbose)
  File "/usr/local/lib/python3.6/site-packages/pip/_internal/req/req_uninstall.py", line 388, in remove
    moved.stash(path)
  File "/usr/local/lib/python3.6/site-packages/pip/_internal/req/req_uninstall.py", line 277, in stash
    renames(path, new_path)
  File "/usr/local/lib/python3.6/site-packages/pip/_internal/utils/misc.py", line 402, in renames
    shutil.move(old, new)
  File "/usr/local/lib/python3.6/shutil.py", line 562, in move
    rmtree(src)
  File "/usr/local/lib/python3.6/shutil.py", line 490, in rmtree
    onerror(os.rmdir, path, sys.exc_info())
  File "/usr/local/lib/python3.6/shutil.py", line 488, in rmtree
    os.rmdir(path)
OSError: [Errno 22] Invalid argument: '/usr/local/lib/python3.6/site-packages/requests-2.22.0.dist-info/'
The command '/bin/sh -c pip uninstall requests --yes' returned a non-zero code: 2

tocosastalo avatar Aug 28 '19 12:08 tocosastalo

When I change dockerfile to:

FROM python:3.6

WORKDIR /app

RUN pip install requests
RUN pip install --force-reinstall requests

Then the error is: ERROR: Could not install packages due to an EnvironmentError: [Errno 22] Invalid argument

Output

Step 1/4 : FROM python:3.6
 ---> 1c515a624542
Step 2/4 : WORKDIR /app
 ---> Running in 03c2a2083714
Removing intermediate container 03c2a2083714
 ---> e351dc6b1e13
Step 3/4 : RUN pip install requests
 ---> Running in e171d496c0bb
Collecting requests
  Downloading https://files.pythonhosted.org/packages/51/bd/23c926cd341ea6b7dd0b2a00aba99ae0f828be89d72b2190f27c11d4b7fb/requests-2.22.0-py2.py3-none-any.whl (57kB)
Collecting certifi>=2017.4.17 (from requests)
  Downloading https://files.pythonhosted.org/packages/69/1b/b853c7a9d4f6a6d00749e94eb6f3a041e342a885b87340b79c1ef73e3a78/certifi-2019.6.16-py2.py3-none-any.whl (157kB)
Collecting urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 (from requests)
  Downloading https://files.pythonhosted.org/packages/e6/60/247f23a7121ae632d62811ba7f273d0e58972d75e58a94d329d51550a47d/urllib3-1.25.3-py2.py3-none-any.whl (150kB)
Collecting idna<2.9,>=2.5 (from requests)
  Downloading https://files.pythonhosted.org/packages/14/2c/cd551d81dbe15200be1cf41cd03869a46fe7226e7450af7a6545bfc474c9/idna-2.8-py2.py3-none-any.whl (58kB)
Collecting chardet<3.1.0,>=3.0.2 (from requests)
  Downloading https://files.pythonhosted.org/packages/bc/a9/01ffebfb562e4274b6487b4bb1ddec7ca55ec7510b22e4c51f14098443b8/chardet-3.0.4-py2.py3-none-any.whl (133kB)
Installing collected packages: certifi, urllib3, idna, chardet, requests
Successfully installed certifi-2019.6.16 chardet-3.0.4 idna-2.8 requests-2.22.0 urllib3-1.25.3
Removing intermediate container e171d496c0bb
 ---> 3c1e5e0bd2d5
Step 4/4 : RUN pip install --force-reinstall requests
 ---> Running in c521f3e734e4
Collecting requests
  Using cached https://files.pythonhosted.org/packages/51/bd/23c926cd341ea6b7dd0b2a00aba99ae0f828be89d72b2190f27c11d4b7fb/requests-2.22.0-py2.py3-none-any.whl
Collecting idna<2.9,>=2.5 (from requests)
  Using cached https://files.pythonhosted.org/packages/14/2c/cd551d81dbe15200be1cf41cd03869a46fe7226e7450af7a6545bfc474c9/idna-2.8-py2.py3-none-any.whl
Collecting chardet<3.1.0,>=3.0.2 (from requests)
  Using cached https://files.pythonhosted.org/packages/bc/a9/01ffebfb562e4274b6487b4bb1ddec7ca55ec7510b22e4c51f14098443b8/chardet-3.0.4-py2.py3-none-any.whl
Collecting certifi>=2017.4.17 (from requests)
  Using cached https://files.pythonhosted.org/packages/69/1b/b853c7a9d4f6a6d00749e94eb6f3a041e342a885b87340b79c1ef73e3a78/certifi-2019.6.16-py2.py3-none-any.whl
Collecting urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 (from requests)
  Using cached https://files.pythonhosted.org/packages/e6/60/247f23a7121ae632d62811ba7f273d0e58972d75e58a94d329d51550a47d/urllib3-1.25.3-py2.py3-none-any.whl
Installing collected packages: idna, chardet, certifi, urllib3, requests
  Found existing installation: idna 2.8
    Uninstalling idna-2.8:
ERROR: Could not install packages due to an EnvironmentError: [Errno 22] Invalid argument: '/usr/local/lib/python3.6/site-packages/idna-2.8.dist-info/'

tocosastalo avatar Aug 28 '19 12:08 tocosastalo

Interesting... os.rename cannot be used across mount points, that makes sense. The reason seems to be that requests are installed in one image layer and un- or re-installed in another, by which time the original image layer is effectively read-only / a different file system.

I'm a bit confused by rename paths:

  • '/usr/local/lib/python3.6/site-packages/requests-2.22.0.dist-info/' ->
  • '/usr/local/lib/python3.6/site-packages/~equests-2.22.0.dist-info'

That tilda is just odd...

dimaqq avatar Aug 29 '19 06:08 dimaqq

Btw., the very same Dockerfile as OP, against same base image 1c515a624542, builds just fine with Docker for Mac:

Server: Docker Engine - Community
 Engine:
  Version:          19.03.1

Perhaps upgrade your docker (server) or change storage driver (maybe) or drop CentOS if the earlier are not possible?

dimaqq avatar Aug 29 '19 06:08 dimaqq

Can you please provide the output of docker info for the system on which you're having the issue? With that we should be able to narrow down to particular storage driver, filesystem, or plugins that may be causing this.

chrahunt avatar Aug 31 '19 00:08 chrahunt

Thank you for your responses. Yes, paths renaming is very strange for me too. Unfortunately, now I don't have permissions to update docker or centos, so i can't try these options.

docker info output:

Containers: 10
 Running: 0
 Paused: 0
 Stopped: 10
Images: 27
Server Version: 18.09.7
Storage Driver: overlay
 Backing Filesystem: extfs
 Supports d_type: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: bridge host macvlan null overlay
 Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 894b81a4b802e4eb2a91d1ce216b8817763c29fb
runc version: 425e105d5a03fabd737a126ad93d62a9eeede87f
init version: fec3683
Security Options:
 seccomp
  Profile: default
Kernel Version: 3.10.0-229.7.2.el7.x86_64
Operating System: CentOS Linux 7 (Core)
OSType: linux
Architecture: x86_64
CPUs: 4
Total Memory: 7.64GiB
Name: ****
ID: RN7W:WMTY:OSLA:4CUM:EMB5:QOFC:QSZ6:5R3B:5QKR:2I7E:FSH5:FO2F
Docker Root Dir: /docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: false
Product License: Community Engine

WARNING: bridge-nf-call-iptables is disabled
WARNING: bridge-nf-call-ip6tables is disabled
WARNING: the overlay storage-driver is deprecated, and will be removed in a future release.

tocosastalo avatar Sep 02 '19 14:09 tocosastalo

I'm a bit confused by rename paths:

* `'/usr/local/lib/python3.6/site-packages/requests-2.22.0.dist-info/'` ->

* `'/usr/local/lib/python3.6/site-packages/~equests-2.22.0.dist-info'`

That tilda is just odd...

Did you managa to find out where this tilde is comming from?

I am seeing this spuriously in another build and cannot make any sense of it. :confused:

cfra avatar Jul 19 '21 11:07 cfra 

>>> bin(ord("~"))
'0b1111110'
>>> bin(ord("r"))
'0b1110010'

That's only a 2-bit difference... still something tells me that [perhaps] it's not a hardware error; I think some silly logic to name a temporary file unique-ish by replacing the first character with a tilda is [perhaps] more likely...

dimaqq avatar Jul 19 '21 12:07 dimaqq

Thank you for your prompt answer. :+1:

I figured out where the tilde is coming from and your assumption is correct:

Pip has a class AdjacentTempDir that it apparently uses to shuffle things around.

So access to the path with tilde is indeed expected and correct, even if it looks odd at first glance.

In my case, the spurious failures are apparently caused by Poetry running multiple pip instances in parallel in a way that is unsafe, so it is not really related to the issue here, although it looked similar at first, so I'd like to apologize for lowering this issue's SNR.

cfra avatar Jul 19 '21 14:07 cfra