Error response from daemon: open /var/lib/docker/overlay2/XXX: no such file or directory

[tomas-wood]

• [x] This is a bug report
• [ ] This is a feature request
• [x] I searched existing issues before opening this one

Expected behavior

docker push <tag> should push the given image to dockerhub. docker save -o <tag>.tar <tag> should save the image to a tarball.

Actual behavior

docker push <tag> doesn't push the image to dockerhub. I get an error saying

Error response from daemon: open /var/lib/docker/overlay2/blah/blah/blah/dir_000027_000001.xhtml: no such file or directory

same error for docker save

Steps to reproduce the behavior

Try to build a caffe2 image. I double dog dare you.

Output of docker version:

Client:
 Version:           18.06.1-ce
 API version:       1.38
 Go version:        go1.10.3
 Git commit:        e68fc7a
 Built:             Tue Aug 21 17:24:56 2018
 OS/Arch:           linux/amd64
 Experimental:      false

Server:
 Engine:
  Version:          18.06.1-ce
  API version:      1.38 (minimum version 1.12)
  Go version:       go1.10.3
  Git commit:       e68fc7a
  Built:            Tue Aug 21 17:23:21 2018
  OS/Arch:          linux/amd64
  Experimental:     false

Output of docker info:

Containers: 3
 Running: 1
 Paused: 0
 Stopped: 2
Images: 36
Server Version: 18.06.1-ce
Storage Driver: overlay2
 Backing Filesystem: extfs
 Supports d_type: true
 Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: bridge host macvlan null overlay
 Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog
Swarm: inactive
Runtimes: runc nvidia
Default Runtime: runc
Init Binary: docker-init
containerd version: 468a545b9edcd5932818eb9de8e72413e616e86e
runc version: 69663f0bd4b60df09991c08812a60108003fa340
init version: fec3683
Security Options:
 apparmor
 seccomp
  Profile: default
Kernel Version: 4.4.0-134-generic
Operating System: Ubuntu 16.04.5 LTS
OSType: linux
Architecture: x86_64
CPUs: 24
Total Memory: 125.8GiB
Name: fw0013591
ID: FJ2J:H5RI:QJBJ:NJYO:VJJ6:7XIU:SIWB:KGNC:56MK:7PT7:TWXF:UZQX
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Username: odellus
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: false

WARNING: No swap limit support

Additional environment details (AWS, VirtualBox, physical, etc.)

[tahirhsn91]

@huawei-tomas I have tried to reproduce this error with docker-ce version "18.06.1-ce" but I am unable to reproduce it. Can you please provide more details about the steps you have performed.

Its may be the issue with docker update in your system, you might have lost some file while installing or updating docker. Did try to uninstall docker and install it again?

[gdf8gdn8]

I can reproduce the error.

DISTRIB_ID=ManjaroLinux
DISTRIB_RELEASE=18.0.2
DISTRIB_CODENAME=Illyria
DISTRIB_DESCRIPTION="Manjaro Linux"

I have to remove docker installation and clean /var/lib/docker folder. Reboot and install docker again. Enable docker (i.e. systemctl) and reboot again.

[kisshore]

Event it is same in

# docker version
Client:
 Version:           18.06.3-ce
 API version:       1.38
 Go version:        go1.10.3
 Git commit:        d7080c1
 Built:             Wed Feb 20 02:26:51 2019
 OS/Arch:           linux/amd64
 Experimental:      false

Server:
 Engine:
  Version:          18.06.3-ce
  API version:      1.38 (minimum version 1.12)
  Go version:       go1.10.3
  Git commit:       d7080c1
  Built:            Wed Feb 20 02:28:17 2019
  OS/Arch:          linux/amd64
  Experimental:     false

[Wassap124]

sudo service restart docker

did the trick for me, hope it helps

[adminjl0]

Has anyone solved this yet? I can't push image to Docker Hub and got error.

After I prune system I re-downloaded older/same image but it failed.

That is strange.

[adminjl0]

sudo service restart docker

did the trick for me, hope it helps

Didn't work even after restart and reboot.

[fboulay]

docker system prune -af worked for me

[xiaojueguan]

I fell into this problem because i tried to mv /var/lib/docker/overlay2 to some other place with docker starting. And docker system prune -af works for me.

[pursuezhao]

docker system prune -af worked for me, thanks!

[elyess78]

@fboulay you made my day , thanks!!

[vishalk663]

docker system prune -af worked for me, thanks

[qqizai]

docker system prune -af

this command worked for me，thanks.

[kangweikiat]

docker system prune -af worked, thanks !!

[j143]

~docker system prune -af~

⚠️ Do not run this option in Production clusters. Only prune the required things. Sometimes simply stopping and starting the docker service resolves this issue.

If you really want to use the above command, please do not use -f flag.

I suggest to spending some time understanding this docker system prune command. It literally removes all the loaded images and makes the cluster unusable. Please investigate what is causing this issue in your cluster by checking system events, logs etc. and resolve the issue.

https://docs.docker.com/engine/reference/commandline/system_prune/

[karuppasamyit10]

sudo service restart docker

did the trick for me, hope it helps

sudo systemctl restart docker

This is works for me in ec2 linux

[orneladjouhou]

docker system prune -af

thanks! It works for me too

[channakesa22]

sudo service restart docker

did the trick for me, hope it helps

sudo systemctl restart docker

This is works for me in ec2 linux

Hello @karuppasamyit10 Even i use Azure machines, can you please tell me so if i use this command sudo systemctl restart docker- does we need to start container& layers also?

1. if i use this command does docker cache also will clear?

[laurensk]

In my case I just hadn't enough free disk space...

[AtharvaTilak]

docker system prune -af worked for me

Thanks worked for me too

[Gui-mp8]

docker system prune -af Works!!

[dev1729anurag]

remove docker completely https://askubuntu.com/questions/935569/how-to-completely-uninstall-docker and then install it again https://docs.docker.com/engine/install/ubuntu/ In my case, I by mistake removed that file which then later started giving me this error.

[CGAlei]

Error response from daemon: open /var/lib/docker/volumes/paperless_redisdata/_data: no such file or directory

[umbertix]

We are having the same issue, so far pruning the files is giving us a 60% success rate. Some times we need to trigger it multiples times. Does anyone know the underlying reason for this error? So far my google foo has not yearned any good results, except many people running into the same problem but without a definitive fix.

[vadimbukstein]

Hi All, I faced the same issue on CentOS 7 with Docker server version 18.09.7 when executed command "docker image inspect <image_id>". Other commands (pull, run) were working fine with no issues. There are the steps that solved my issue:

1. Remove all docker containers and images: docker stop $(docker ps -a -q) docker rm $(docker ps -a -q) docker rmi $(docker images -a -q)
2. Uninstall Docker yum remove -y docker \ docker-client \ docker-client-latest \ docker-common \ docker-latest \ docker-latest-logrotate \ docker-logrotate \ docker-selinux \ docker-engine-selinux \ docker-engine
3. Delete all Docker related resources rm -rf /run/docker rm -rf /var/run/docker rm -rf /etc/docker rm -rf /var/lib/docker
4. Install Docker as per official documentation : https://docs.docker.com/engine/install/centos/#install-using-the-repository
5. Restart docker service systemctl restart docker Before executing the above steps I have tried docker system prune -a -f , remove /var/lib/docker/overlay2 , reboot docker service and the server without any success. The steps I mentioned above has resolved my issue.

[zhujintao]

docker system prune -af worked for me, thanks!

[Kkbrothers8795]

It's because you have deleted images path.

[Squawkykaka]

docker system prune -af worked for me after i deleted the /var/lib/docker/overlay2/429963a5816afd53c1450bd8b15fa8bda2db278423f3eeccc8647cea6e664fa6 directory