Impossible to pull Home Assistant 2023.10.X

[ghost58400]

trafficstars

Describe the issue you are experiencing

Impossible to update Home Assistant from 2023.9.3 to 2023.10.X, everything else is up to date.

What operating system image do you use?

ova (for Virtual Machines)

What version of Home Assistant Operating System is installed?

11.0

Did you upgrade the Operating System.

Yes

Steps to reproduce the issue

Go to Settings, Home Assistant Core Update, and click Install

Anything in the Supervisor logs that might be useful for us?

Here is the supervisor logs :

23-10-17 15:03:34 INFO (MainThread) [supervisor.homeassistant.core] Updating Home Assistant to version 2023.10.3
23-10-17 15:03:34 INFO (MainThread) [supervisor.docker.interface] Updating image ghcr.io/home-assistant/qemux86-64-homeassistant:2023.9.3 to ghcr.io/home-assistant/qemux86-64-homeassistant:2023.10.3
23-10-17 15:03:34 INFO (MainThread) [supervisor.docker.interface] Downloading docker image ghcr.io/home-assistant/qemux86-64-homeassistant with tag 2023.10.3.
23-10-17 15:03:48 ERROR (MainThread) [supervisor.docker.interface] Can't install ghcr.io/home-assistant/qemux86-64-homeassistant:2023.10.3: 404 Client Error for http+docker://localhost/v1.43/images/ghcr.io/home-assistant/qemux86-64-homeassistant:2023.10.3/json: Not Found ("No such image: ghcr.io/home-assistant/qemux86-64-homeassistant:2023.10.3")
23-10-17 15:03:48 WARNING (MainThread) [supervisor.homeassistant.core] Updating Home Assistant image failed

Anything in the Host logs that might be useful for us?

Here is Host logs :

Oct 17 13:03:35 hass kernel: overlayfs: overlapping lowerdir path
Oct 17 13:03:35 hass dockerd[460]: time="2023-10-17T13:03:35.803896586Z" level=error msg="error unmounting /mnt/data/docker/overlay2/ab41b898c1289c3409be5cad51f8fb58d95033b30096b34824d84fbf26b6dc23/merged: invalid argument" storage-driver=overlay2
Oct 17 13:03:42 hass dockerd[460]: time="2023-10-17T13:03:42.904804447Z" level=info msg="Attempting next endpoint for pull after error: failed to register layer: error creating overlay mount to /mnt/data/docker/overlay2/ab41b898c1289c3409be5cad51f8fb58d95033b30096b34824d84fbf26b6dc23/merged: too many levels of symbolic links"

System information

System Information

version	core-2023.9.3
installation_type	Home Assistant OS
dev	false
hassio	true
docker	true
user	root
virtualenv	false
python_version	3.11.5
os_name	Linux
os_version	6.1.56
arch	x86_64
timezone	Europe/Paris
config_dir	/config

Home Assistant Community Store

GitHub API	ok
GitHub Content	ok
GitHub Web	ok
GitHub API Calls Remaining	5000
Installed Version	1.33.0
Stage	running
Available Repositories	1395
Downloaded Repositories	37
HACS Data	ok

Home Assistant Cloud

logged_in	false
can_reach_cert_server	ok
can_reach_cloud_auth	ok
can_reach_cloud	ok

Home Assistant Supervisor

host_os	Home Assistant OS 11.0
update_channel	stable
supervisor_version	supervisor-2023.10.0
agent_version	1.6.0
docker_version	24.0.6
disk_total	30.8 GB
disk_used	25.3 GB
healthy	true
supported	true
board	ova
supervisor_api	ok
version_api	ok
installed_addons	File editor (5.6.0), Home Assistant Google Drive Backup (0.111.1), Studio Code Server (5.12.0), Advanced SSH & Web Terminal (15.1.0), phpMyAdmin (0.8.9), Samba share (10.0.2), motionEye (0.19.1), Mosquitto broker (6.3.1), RTSPtoWeb - WebRTC (1.4.0), Guacamole (1.3.0), MariaDB (2.6.1), ESPHome (2023.9.3), go2rtc (1.8.1), Matter Server (4.9.0), LibreTiny ESPHome (2023.9.0-dev), Tailscale (0.12.0), MyElectricalData (0.9.2), Free Games Claimer (NoVNC not working) (1.4-test22)

Dashboards

dashboards	3
resources	18
views	11
mode	storage

Recorder

oldest_recorder_run	15 octobre 2023 à 17:04
current_recorder_run	16 octobre 2023 à 14:18
estimated_db_size	702.17 MiB
database_engine	mysql
database_version	10.6.12

Sonoff

version	3.5.2 (2ad1cd7)
cloud_online	1 / 2
local_online	1 / 1

Xiaomi Miot Auto

component_version	0.7.13
can_reach_server	ok
can_reach_spec	ok
logged_accounts	1
total_devices	1

Additional information

Same error appears if I try docker pull ghcr.io/home-assistant/qemux86-64-homeassistant:2023.10.3 on HassOS through SSH I can still pull other versions of the Home Assistant image without problem such as 2023.09.2, the problem appears only with 2023.10.X

Tried docker pull ghcr.io/home-assistant/qemux86-64-homeassistant:2023.10.3 on an other linux machine, pulled without problem, so it is linked to my HassOS machine.

[JaCoLJcL]

Same for my raspberry pi 4 unit... :/

[agners]

This seems like some corruption on the storage.

Can you try this command from the console?

ha supervisor repair

[ghost58400]

# ha supervisor repair
Command completed successfully.

The command completes successfully but still impossible to apply the update with same error

[perrucon]

Same here, RPI2

[ghost58400]

Found a solution here : https://stackoverflow.com/questions/60209172/docker-cannot-delete-intermediate-images-from-a-broken-pull

Here is what I did :

• Make a backup
• Connect to HassOS via SSH
• systemctl stop docker docker.socket
• rm -rf /var/lib/docker (you can ignore the error message)
• reboot now

HassOS will fix itself automatically on startup, installing the latest Home Assistant version, after full startup and auto-repair, you might need to do ha supervisor repair and reboot after completion, you might also need to re-configure automatic startup of addons

This issue appears when you shutdown forcefully HassOS during an update (docker pull), because a layer gets corrupted.

It is not a great fix but it worked for me, maybe a similar procedure should be implemented in supervisor repair to clear this type of issue

[JaCoLJcL]

Found a solution here : https://stackoverflow.com/questions/60209172/docker-cannot-delete-intermediate-images-from-a-broken-pull

Here is what I did :

• Make a backup
• Connect to HassOS via SSH
• systemctl stop docker docker.socket
• rm -rf /var/lib/docker (you can ignore the error message)
• reboot now

HassOS will fix itself automatically on startup, installing the latest Home Assistant version, after full startup and auto-repair, you might need to do ha supervisor repair and reboot after completion, you might also need to re-configure automatic startup of addons

This issue appears when you shutdown forcefully HassOS during an update (docker pull), because a layer gets corrupted.

It is not a great fix but it worked for me, maybe a similar procedure should be implemented in supervisor repair to clear this type of issue

That sounds like a possible HA failure. Can these actions broke my system? I use a raspberry pi 4

[ghost58400]

Since I made a backup, in the worst case I would have reinstalled HassOS and restore the backup. It worked flawlessly for me but I cannot guarantee that it will work for everyone.

[JaCoLJcL]

@ghost58400 ok i see...

Please tell me how to get this supervisor logs before restart after download image?

[ghost58400]

You do not need to get anything, if everything is working after the first reboot you do not need the second part. If you have problems with addons after the first reboot, you need to do the repair and reboot.

[EDIflyer]

Same issue here trying to go from 2023.10.3 to 2023.10.5 - will try rebooting.

23-10-25 11:49:02 INFO (MainThread) [supervisor.backups.manager] Backup 9160be73 starting stage addon_repositories
23-10-25 11:49:02 INFO (MainThread) [supervisor.backups.manager] Backup 9160be73 starting stage docker_config
23-10-25 11:49:02 INFO (MainThread) [supervisor.backups.manager] Creating new partial backup with slug 9160be73
23-10-25 11:49:02 INFO (MainThread) [supervisor.backups.manager] Backup 9160be73 starting stage home_assistant
23-10-25 11:49:02 INFO (MainThread) [supervisor.homeassistant.module] Backing up Home Assistant Core config folder
23-10-25 11:49:24 INFO (MainThread) [supervisor.homeassistant.module] Backup Home Assistant Core config folder done
23-10-25 11:49:24 INFO (MainThread) [supervisor.backups.manager] Backup 9160be73 starting stage finishing_file
23-10-25 11:49:24 INFO (MainThread) [supervisor.backups.manager] Creating partial backup with slug 9160be73 completed
23-10-25 11:49:24 INFO (MainThread) [supervisor.homeassistant.core] Updating Home Assistant to version 2023.10.5
23-10-25 11:49:24 INFO (MainThread) [supervisor.docker.interface] Updating image ghcr.io/home-assistant/raspberrypi4-homeassistant:2023.10.3 to ghcr.io/home-assistant/raspberrypi4-homeassistant:2023.10.5
23-10-25 11:49:24 INFO (MainThread) [supervisor.docker.interface] Downloading docker image ghcr.io/home-assistant/raspberrypi4-homeassistant with tag 2023.10.5.
23-10-25 11:49:24 ERROR (MainThread) [supervisor.docker.interface] Can't install ghcr.io/home-assistant/raspberrypi4-homeassistant:2023.10.5: 500 Server Error for http+docker://localhost/v1.43/images/create?tag=2023.10.5&fromImage=ghcr.io%2Fhome-assistant%2Fraspberrypi4-homeassistant&platform=linux%2Farm%2Fv7: Internal Server Error ("Get "https://ghcr.io/v2/": EOF")
23-10-25 11:49:24 WARNING (MainThread) [supervisor.homeassistant.core] Updating Home Assistant image failed

[agners]

@EDIflyer @perrucon do you see the same issue in the host logs OP is seeing (too many levels of symbolic links)?

This is likely caused by some file system corruption or power outage. It seems that Docker recently applied a fix for this problem: https://github.com/moby/moby/pull/46703.

@ghost58400 can you run the command after a reboot, and then check the Supervisor logs?

[ghost58400]

Hello @agners

For my case, I am 99% sure that the issue was caused by a power outage because the HassOS vm has been shut down during the download of Home Assistant update, so it corresponds perfectly to the pull-request that you linked.

Unfortunately for the progress of the issue, I cannot reproduce it anymore because I applied the procedure explained in my previous comment which solved the issue :

Found a solution here : https://stackoverflow.com/questions/60209172/docker-cannot-delete-intermediate-images-from-a-broken-pull

Here is what I did :

• Make a backup
• Connect to HassOS via SSH
• systemctl stop docker docker.socket
• rm -rf /var/lib/docker (you can ignore the error message)
• reboot now

HassOS will fix itself automatically on startup, installing the latest Home Assistant version, after full startup and auto-repair, you might need to do ha supervisor repair and reboot after completion, you might also need to re-configure automatic startup of addons

This issue appears when you shutdown forcefully HassOS during an update (docker pull), because a layer gets corrupted.

It is not a great fix but it worked for me, maybe a similar procedure should be implemented in supervisor repair to clear this type of issue

[hostops]

Hi! Same problem here. I cannot perform clean official install on rpi3 I believe this guy has the same issue: https://community.home-assistant.io/t/first-time-install-not-working/626061

[EDIflyer]

@EDIflyer @perrucon do you see the same issue in the host logs OP is seeing (too many levels of symbolic links)?

This is likely caused by some file system corruption or power outage. It seems that Docker recently applied a fix for this problem: moby/moby#46703.

Sorry for delay in replying - just did sudo journalctl -g "too many levels of symbolic links" on the host and nothing showing up. I've updated docker anyway:

Client: Docker Engine - Community
 Version:           24.0.7
 API version:       1.43
 Go version:        go1.20.10
 Git commit:        afdd53b
 Built:             Thu Oct 26 09:08:26 2023
 OS/Arch:           linux/arm
 Context:           default

Server: Docker Engine - Community
 Engine:
  Version:          24.0.7
  API version:      1.43 (minimum version 1.12)
  Go version:       go1.20.10
  Git commit:       311b9ff
  Built:            Thu Oct 26 09:08:26 2023
  OS/Arch:          linux/arm
  Experimental:     false
 containerd:
  Version:          1.6.24
  GitCommit:        61f9fd88f79f081d64d6fa3bb1a0dc71ec870523
 runc:
  Version:          1.1.9
  GitCommit:        v1.1.9-0-gccaecfc
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

Rebooted and still getting this:

Trying https://ghcr.io/token?scope=repository%3Ahome-assistant%2Fraspberrypi4-homeassistant%3Apull&service=ghcr.io seems to generate a token OK but not sure if that's relevant!

[agners]

@hostops it seems you have a new installation? In that case installing HAOS onto the SD card is probably your best option. If the problem persists, there might be a problem with the SD card.

@EDIflyer can you also try ha supervisor repair?

[CoenGoedhart1]

@agners @hostops I have the same problem while doing a fresh install on my rpi4, tried changing my DNS to 8.8.8.8 and 1.1.1.1 but no succes.

[EDIflyer]

@agners done...

23-10-31 23:57:01 INFO (MainThread) [supervisor.supervisor] Repairing Supervisor 2023.10.1
23-10-31 23:57:01 INFO (MainThread) [supervisor.core] Finished repair of Supervisor Environment

Still no change I'm afraid:

23-10-31 23:57:21 INFO (MainThread) [supervisor.backups.manager] Backup 88dcb4da starting stage addon_repositories
23-10-31 23:57:21 INFO (MainThread) [supervisor.backups.manager] Backup 88dcb4da starting stage docker_config
23-10-31 23:57:21 INFO (MainThread) [supervisor.backups.manager] Creating new partial backup with slug 88dcb4da
23-10-31 23:57:21 INFO (MainThread) [supervisor.backups.manager] Backup 88dcb4da starting stage home_assistant
23-10-31 23:57:22 INFO (MainThread) [supervisor.homeassistant.module] Backing up Home Assistant Core config folder
23-10-31 23:57:45 INFO (MainThread) [supervisor.homeassistant.module] Backup Home Assistant Core config folder done
23-10-31 23:57:45 INFO (MainThread) [supervisor.backups.manager] Backup 88dcb4da starting stage finishing_file
23-10-31 23:57:45 INFO (MainThread) [supervisor.backups.manager] Creating partial backup with slug 88dcb4da completed
23-10-31 23:57:45 INFO (MainThread) [supervisor.homeassistant.core] Updating Home Assistant to version 2023.10.5
23-10-31 23:57:45 INFO (MainThread) [supervisor.docker.interface] Updating image ghcr.io/home-assistant/raspberrypi4-homeassistant:2023.10.3 to ghcr.io/home-assistant/raspberrypi4-homeassistant:2023.10.5
23-10-31 23:57:45 INFO (MainThread) [supervisor.docker.interface] Downloading docker image ghcr.io/home-assistant/raspberrypi4-homeassistant with tag 2023.10.5.
23-10-31 23:57:46 ERROR (MainThread) [supervisor.docker.interface] Can't install ghcr.io/home-assistant/raspberrypi4-homeassistant:2023.10.5: 500 Server Error for http+docker://localhost/v1.43/images/create?tag=2023.10.5&fromImage=ghcr.io%2Fhome-assistant%2Fraspberrypi4-homeassistant&platform=linux%2Farm%2Fv7: Internal Server Error ("Head "https://ghcr.io/v2/home-assistant/raspberrypi4-homeassistant/manifests/2023.10.5": Get "https://ghcr.io/token?scope=repository%3Ahome-assistant%2Fraspberrypi4-homeassistant%3Apull&service=ghcr.io": EOF")
23-10-31 23:57:46 WARNING (MainThread) [supervisor.homeassistant.core] Updating Home Assistant image failed

[EDIflyer]

I'm still not able to update Core OS, yet was able to update Supervisor and some plugins OK!

23-11-28 15:38:15 INFO (MainThread) [supervisor.homeassistant.core] Updating Home Assistant to version 2023.11.3
23-11-28 15:38:15 INFO (MainThread) [supervisor.docker.interface] Updating image ghcr.io/home-assistant/raspberrypi4-homeassistant:2023.10.3 to ghcr.io/home-assistant/raspberrypi4-homeassistant:2023.11.3
23-11-28 15:38:15 INFO (MainThread) [supervisor.docker.interface] Downloading docker image ghcr.io/home-assistant/raspberrypi4-homeassistant with tag 2023.11.3.
23-11-28 15:40:24 ERROR (MainThread) [supervisor.docker.interface] Can't install ghcr.io/home-assistant/raspberrypi4-homeassistant:2023.11.3: 404 Client Error for http+docker://localhost/v1.43/images/ghcr.io/home-assistant/raspberrypi4-homeassistant:2023.11.3/json: Not Found ("No such image: ghcr.io/home-assistant/raspberrypi4-homeassistant:2023.11.3")
23-11-28 15:40:24 WARNING (MainThread) [supervisor.homeassistant.core] Updating Home Assistant image failed

It seems to be redownloading it OK each time but could there be something cached that's causing the issue?

[agners]

@EDIflyer hm, it seems that your case changed from EOF to Not Found? :thinking:

Can you check the Host logs after a failed attempt?

[EDIflyer]

@agners yes I'm not sure quite what's going on - I did a flush of dns and eventually seemed to get it to install - the weird thing was Supervisor updated without a problem, not sure if it was some weird transient issue. I guess I'll see what happens with the next HA update 😬

[github-actions[bot]]

There hasn't been any activity on this issue recently. To keep our backlog manageable we have to clean old issues, as many of them have already been resolved with the latest updates. Please make sure to update to the latest Home Assistant OS version and check if that solves the issue. Let us know if that works for you by adding a comment 👍 This issue has now been marked as stale and will be closed if no further activity occurs. Thank you for your contributions.