robweber
Creating a Backup with some Docker images will fail (get stuck).
Hi Rob,
Understood from your last comment in the Backup thread on the Kodi forum that I better should create an issue here.
I'm having this issue on my Atom X5 based 'Tronsmart Ara X5 Plus' with LibreELEC and some Docker images. I'm able to reproduce it on a RPi3 using Escalade own build, which include (scripts to download) an Docker image.
From debug it appears to just get stuck (last few lines of kode.log) with a file named fifo:
19:30:52.640 T:1854927760 DEBUG: script.xbmcbackup-1.0.9: Writing file: /storage/.kodi/userdata/addon_data/service.system.docker/docker/overlay2/bff94681006d543fa14cce62e0f806de78b9fb8d65894b054d2ca9c31844ef94/diff/var/run/systemd/journal/stdout
19:30:52.641 T:1854927760 DEBUG: script.xbmcbackup-1.0.9: Writing file: /storage/.kodi/userdata/addon_data/service.system.docker/docker/overlay2/bff94681006d543fa14cce62e0f806de78b9fb8d65894b054d2ca9c31844ef94/diff/var/run/systemd/journal/dev-log
19:30:52.642 T:1854927760 DEBUG: script.xbmcbackup-1.0.9: Writing file: -/storage/.kodi/userdata/addon_data/service.system.docker/docker/overlay2/bff94681006d543fa14cce62e0f806de78b9fb8d65894b054d2ca9c31844ef94/diff/var/run/systemd/ask-password
19:30:52.642 T:1854927760 DEBUG: script.xbmcbackup-1.0.9: Writing file: -/storage/.kodi/userdata/addon_data/service.system.docker/docker/overlay2/bff94681006d543fa14cce62e0f806de78b9fb8d65894b054d2ca9c31844ef94/diff/var/run/systemd/transient
19:30:52.642 T:1854927760 DEBUG: script.xbmcbackup-1.0.9: Writing file: -/storage/.kodi/userdata/addon_data/service.system.docker/docker/overlay2/bff94681006d543fa14cce62e0f806de78b9fb8d65894b054d2ca9c31844ef94/diff/var/run/systemd/inaccessible
19:30:52.642 T:1854927760 DEBUG: script.xbmcbackup-1.0.9: Writing file: -/storage/.kodi/userdata/addon_data/service.system.docker/docker/overlay2/bff94681006d543fa14cce62e0f806de78b9fb8d65894b054d2ca9c31844ef94/diff/var/run/systemd/inaccessible/dir
19:30:52.642 T:1854927760 DEBUG: script.xbmcbackup-1.0.9: Writing file: /storage/.kodi/userdata/addon_data/service.system.docker/docker/overlay2/bff94681006d543fa14cce62e0f806de78b9fb8d65894b054d2ca9c31844ef94/diff/var/run/systemd/inaccessible/sock
19:30:52.643 T:1854927760 DEBUG: script.xbmcbackup-1.0.9: Writing file: /storage/.kodi/userdata/addon_data/service.system.docker/docker/overlay2/bff94681006d543fa14cce62e0f806de78b9fb8d65894b054d2ca9c31844ef94/diff/var/run/systemd/inaccessible/fifo
19:31:07.074 T:1911550864 DEBUG: CAESinkPi:Drain delay:99ms now:0ms
19:31:17.075 T:1911550864 DEBUG: CAESinkPi:Deinitialize
19:31:17.075 T:1911550864 DEBUG: CAESinkPi:SetAudioProps hdmi_stream_channels 0 hdmi_channel_map 00000000
19:31:17.083 T:1911550864 DEBUG: COMXCoreComponent::Deinitialize : OMX.broadcom.audio_render handle 0x72002cc0
19:33:07.251 T:1942557584 DEBUG: CAnnouncementManager - Announcement: OnScreensaverActivated from xbmc
19:33:07.251 T:1942557584 DEBUG: GOT ANNOUNCEMENT, type: 4, from xbmc, message OnScreensaverActivated
19:33:07.255 T:1942571392 DEBUG: ------ Window Init () ------
The temporary file is also not growing anymore:
~/.kodi/temp # ls -la
total 54070
drwxr-xr-x 4 root root 1024 Nov 19 19:30 .
drwxr-xr-x 7 root root 1024 Nov 19 19:16 ..
drwxr-xr-x 2 root root 1024 Nov 19 19:16 archive_cache
-rw-r--r-- 1 root root 6925802 Nov 19 19:33 kodi.log
drwxr-xr-x 2 root root 1024 Nov 19 19:16 temp
-rw-r--r-- 1 root root 48435282 Nov 19 19:30 xbmc_backup_temp.zip
~/.kodi/temp # date
Sun Nov 19 19:41:24 UTC 2017
~/.kodi/temp #
If I use the remote I do see new debug messages, so Kodi has not crashed.
19:33:07.255 T:1942571392 DEBUG: ------ Window Init () ------
19:42:42.300 T:1786373008 DEBUG: PushCecKeypress - received key b duration 0 (rep:0 size:0)
19:42:42.300 T:1786373008 DEBUG: PushCecKeypress - added key b
19:42:42.325 T:1942571392 DEBUG: OnKey: 11 (0x0b, obc244) pressed, screen saver/dpms woken up
19:42:42.325 T:1942557584 DEBUG: CAnnouncementManager - Announcement: OnScreensaverDeactivated from xbmc
19:42:42.325 T:1942557584 DEBUG: GOT ANNOUNCEMENT, type: 4, from xbmc, message OnScreensaverDeactivated
19:42:42.659 T:1786373008 DEBUG: PushCecKeypress - received key b duration 359 (rep:0 size:0)
19:42:42.659 T:1786373008 DEBUG: PushCecKeypress - ignored key b
19:42:43.360 T:1942571392 DEBUG: ------ Window Deinit () ------
~/.kodi/temp # ls -la
total 54071
drwxr-xr-x 4 root root 1024 Nov 19 19:30 .
drwxr-xr-x 7 root root 1024 Nov 19 19:16 ..
drwxr-xr-x 2 root root 1024 Nov 19 19:16 archive_cache
-rw-r--r-- 1 root root 6926503 Nov 19 19:42 kodi.log
drwxr-xr-x 2 root root 1024 Nov 19 19:16 temp
-rw-r--r-- 1 root root 48435282 Nov 19 19:30 xbmc_backup_temp.zip
~/.kodi/temp # date
Sun Nov 19 19:44:47 UTC 2017
~/.kodi/temp #
The file it seems to get stuck with looks to be fifio. In the 'xbmc_backup_temp.zip' file I only see the 0 size 'sock' file.:
~/.kodi # cd userdata/addon_data/service.system.docker/docker/overlay2/bff94681006d543fa14cce62e0f806de78b9fb8d65894b054d2ca9c31844ef94/diff/var/run/systemd/inaccessible/
/run/systemd/inaccessible # ls -la
total 0
d--------- 3 root root 160 Oct 6 11:18 .
drwxr-xr-x 15 root root 360 Nov 19 19:20 ..
b--------- 1 root root 0, 0 Oct 6 11:18 blk
c--------- 1 root root 0, 0 Oct 6 11:18 chr
d--------- 2 root root 40 Oct 6 11:18 dir
p--------- 1 root root 0 Oct 6 11:18 fifo
---------- 1 root root 0 Oct 6 11:18 reg
s--------- 1 root root 0 Oct 6 11:18 sock
/run/systemd/inaccessible #
Reproduce the issue on a RPi3:
- Installed 'LibreELEC-RPi2.arm-8.2-devel-20171118.img.gz' from Google-Drive of Escalade (http://bit.ly/2mVGdBf)
- Install Backup and Docker
- Configure Backup, and create a backup. This should succeed witout issue.
- Via ssh excecute: docker run -it --rm -v /usr/bin/qemu-arm-static:/usr/bin/qemu-arm-static armelbuild/debian:jessie bash (don't mind if it failed, It should pull the images)
# docker run -it --rm -v /usr/bin/qemu-arm-static:/usr/bin/qemu-arm-static armelbuild/debian:jessie bash
Unable to find image 'armelbuild/debian:jessie' locally
jessie: Pulling from armelbuild/debian
7c78e7488df7: Pull complete
a3ed95caeb02: Pull complete
8796be12624a: Pull complete
Digest: sha256:dd3e2ffdceb98d84b6ac3419538e1ce92585543cbb9e1adee163bd33aef11b0e
Status: Downloaded newer image for armelbuild/debian:jessie
docker: Error response from daemon: error while creating mount source path '/usr/bin/qemu-arm-static': mkdir /usr/bin/qemu-arm-static: read-only file system.
- Create another backup. (for me this failed with above behavior)
I have the kodi.log (6.6MB) and the xbmc_backup_temp.zip (46.1MB).
Edit: installed system tools:
# cd userdata/addon_data/service.system.docker/docker/overlay2/bff94681006d543fa14cce62e0f806de78b9fb8d65894b054d2ca9c31844ef94/diff/var/run/systemd/inaccessible/
# file fifo
fifo: fifo (named pipe)
#
Thanks for being so thorough. I'm familiar with Docker only in a high level way - not in everyday use. I did some searching on that "fifo" file though. sounds like it could be a data pipe data back and forth. As such it wouldn't surprise me if the file was locked from reading/writing by other processes.
I think the best use case here is simply to not backup either the entire docker addon data directory, or just exclude that "overlay2/" directory and it's sub directories. Take a look at pull request #117 . Would something like work to your benefit here? Essentialy you could add the full addon_data directory to your backup but then exclude the addon_data/service.system.docker/.
I looked at PR #117 and that would indeed give a workaround to exclude the Docker files to keep Backup going. (I'm also not using Docker that much, just playing with it a little.) Is the LibreELEC backup mechanism different, that it can make a backup of this file?
Docker failed to run the image so don't think the file was in use, but I will have a look if the fifo file shows up with lsof.
Rob,
How are you handling symlinks? Is it possible that you end-up in /run/systemd/inaccessible/ due to the symlink? Because /run/systemd/inaccessible/fifo is actually in use according lsof.
LibreELEC:~/.kodi # cd userdata/addon_data/service.system.docker/docker/overlay2/bff94681006d543fa14cce62e0f806de78b9fb8d65894b054d2ca9c31844ef94/diff/var/run/systemd/inaccessible/
LibreELEC:/run/systemd/inaccessible # pwd
/storage/.kodi/userdata/addon_data/service.system.docker/docker/overlay2/bff94681006d543fa14cce62e0f806de78b9fb8d65894b054d2ca9c31844ef94/diff/var/run/systemd/inaccessible
LibreELEC:/run/systemd/inaccessible # cd ..
LibreELEC:/run/systemd # ls -la
total 0
drwxr-xr-x 15 root root 360 Oct 6 11:18 .
drwxr-xr-x 13 root root 380 Oct 6 11:18 ..
drwxr-xr-x 2 root root 40 Oct 6 11:18 ask-password
srwx------ 1 root root 0 Oct 6 11:18 cgroups-agent
drwxr-xr-x 3 root root 60 Oct 6 11:18 generator
d--------- 3 root root 160 Oct 6 11:18 inaccessible
drwxr-xr-x 2 root root 80 Oct 6 11:18 inhibit
drwxr-xr-x 2 root root 60 Oct 6 11:18 initctl
drwxr-xr-x 3 root root 160 Oct 6 11:18 journal
drwxr-xr-x 2 root root 40 Oct 6 11:18 machines
srwxrwxrwx 1 root root 0 Oct 6 11:18 notify
srwxrwxrwx 1 root root 0 Oct 6 11:18 private
drwxr-xr-x 2 root root 60 Oct 6 11:18 seats
drwxr-xr-x 2 root root 40 Oct 6 11:18 sessions
drwxr-xr-x 2 root root 40 Oct 6 11:18 shutdown
drwxr-xr-x 2 root root 40 Oct 6 11:18 system
drwxr-xr-x 2 root root 40 Oct 6 11:18 transient
drwxr-xr-x 2 root root 40 Oct 6 11:18 users
LibreELEC:/run/systemd # pwd
/storage/.kodi/userdata/addon_data/service.system.docker/docker/overlay2/bff94681006d543fa14cce62e0f806de78b9fb8d65894b054d2ca9c31844ef94/diff/var/run/systemd
LibreELEC:/run/systemd # cd ..
LibreELEC:/run # pwd
/storage/.kodi/userdata/addon_data/service.system.docker/docker/overlay2/bff94681006d543fa14cce62e0f806de78b9fb8d65894b054d2ca9c31844ef94/diff/var/run
LibreELEC:/run # ls -la
total 4860
drwxr-xr-x 13 root root 380 Oct 6 11:18 .
drwxr-xr-x 13 root root 226 Nov 18 11:38 ..
drwxr-xr-x 2 root root 80 Oct 6 11:18 connman
-rw-r--r-- 1 root root 4 Oct 6 11:18 crond.pid
drwx------ 4 root root 80 Oct 6 11:18 docker
-r--r--r-- 1 root root 4965246 Oct 6 11:18 hwdb.bin
drwxr-xr-x 2 root root 40 Oct 6 11:18 kodi
drwxr-xr-x 3 root root 80 Oct 6 11:18 libreelec
drwxr-xr-x 2 root root 80 Oct 6 11:18 lirc
drwxr-xr-x 3 root root 60 Oct 6 11:18 log
-rw-r--r-- 1 root root 42 Oct 6 11:18 nologin
srw-rw-rw- 1 root root 0 Oct 6 11:18 rpcbind.sock
drwxr-xr-x 5 root root 200 Oct 6 11:18 samba
drwxr-xr-x 15 root root 360 Oct 6 11:18 systemd
drwxr-xr-x 2 root root 60 Oct 6 11:18 tmpfiles.d
drwxr-xr-x 6 root root 140 Oct 6 11:19 udev
drwxr-xr-x 2 root root 40 Oct 6 11:18 user
-rw-rw-r-- 1 root utmp 0 Oct 6 11:18 utmp
-rw------- 1 root root 0 Oct 6 11:18 xtables.lock
LibreELEC:/run # cd ..
LibreELEC:~/.kodi/userdata/addon_data/service.system.docker/docker/overlay2/bff94681006d543fa14cce62e0f806de78b9fb8d65894b054d2ca9c31844ef94/diff/var # ls -la
total 11
drwxr-xr-x 11 root root 1024 Sep 26 2015 .
drwxr-xr-x 17 root root 1024 Nov 19 19:27 ..
drwxr-xr-x 2 root root 1024 Aug 30 2015 backups
drwxr-xr-x 6 root root 1024 Sep 26 2015 cache
drwxr-xr-x 15 root root 1024 Sep 26 2015 lib
drwxrwsr-x 2 root 50 1024 Aug 30 2015 local
lrwxrwxrwx 1 root root 9 Sep 26 2015 lock -> /run/lock
drwxr-xr-x 4 root root 1024 Sep 26 2015 log
drwxrwsr-x 2 root 8 1024 Sep 26 2015 mail
drwxr-xr-x 2 root root 1024 Sep 26 2015 opt
lrwxrwxrwx 1 root root 4 Sep 26 2015 run -> /run
drwxr-xr-x 4 root root 1024 Sep 26 2015 spool
drwxrwxrwt 2 root root 1024 Aug 30 2015 tmp
LibreELEC:~/.kodi/userdata/addon_data/service.system.docker/docker/overlay2/bff94681006d543fa14cce62e0f806de78b9fb8d65894b054d2ca9c31844ef94/diff/var #
Below the content of /run/systemd/inaccessible/
LibreELEC:/run/systemd/inaccessible # cd /run/systemd/inaccessible/
LibreELEC:/run/systemd/inaccessible # pwd
/run/systemd/inaccessible
LibreELEC:/run/systemd/inaccessible # ls -la
total 0
d--------- 3 root root 160 Oct 6 11:18 .
drwxr-xr-x 15 root root 360 Oct 6 11:18 ..
b--------- 1 root root 0, 0 Oct 6 11:18 blk
c--------- 1 root root 0, 0 Oct 6 11:18 chr
d--------- 2 root root 40 Oct 6 11:18 dir
p--------- 1 root root 0 Oct 6 11:18 fifo
---------- 1 root root 0 Oct 6 11:18 reg
s--------- 1 root root 0 Oct 6 11:18 sock
LibreELEC:/run/systemd/inaccessible # lsof | grep fifo
1 /usr/lib/systemd/systemd /run/systemd/initctl/fifo
LibreELEC:/run/systemd/inaccessible #
Hi Rob,
I know you are working on PR #117, and looking forward to try that out. I also wonder if you have looked at this symlink issue, and if this is something you might be able to solve?
I definitely think you're on to something with the symlink issue. Most of my test systems are Windows for ease of development but I have some RPis but it's just a matter of time to test things out.
I wouldn't be surprised to find that the xbmcvfs file module does follow the symlinks. Definitely haven't forgotten this!
This issue has been automatically marked as inactive because it has not had recent activity. It will be closed if no further activity occurs.
![stale[bot] avatar](https://avatars.githubusercontent.com/in/1724?v=4 "Published by a pub.dev verified publisher"){kind=small}
Ignore the above comment, Probot went crazy and is ignoring the parameters I gave it.
Now i'm run in this same issue on my RPi4 / LibreElec 9.2 Beta 2. latest Backup v1.1.3 hangs on different docker files. Sometime logs, if i delete the logs the on the overlay2 path. Using docker to container easyEPG!
EDIT: Ok, found thats my Backup version completly outdated (v1.1.3). Updating now and exclude docker from backup process! Thanks anyway @robweber for this great addon!
Revisiting this - most of the issues are platform specific and stem from either in use system files being accessed or the Kodi VFS not handling symlinks correctly. In both cases the solution is just to exclude these types of files and directories. The addon is not designed to handle these types of file situations. It's use case is for Kodi specific files that shouldn't be in use.
@robweber Completely agree with closure of this issue. This was opened before the exclude options existed. The exclude options work great avoiding this issue.