overlayfs: upperdir not freed by resync

[Heiko-L]

To reproduce:

1. make a directory, maybe /testasd
2. include that directory in /etc/asd.conf and enable overlay
3. run "asd p" to see that overlayfs size is 0
4. copy a file to /testasd
5. run "asd p" to see that overlayfs size is the size of the copied file
6. wait for systemd timer to trigger or run "sudo /usr/bin/anything-sync-daemon resync" manually
7. run "asd p" again and wonder why overlayfs size has not changed to 0

Maybe I don't understand it, but shouldn't everything be copied from the upper to the lower dir after the timer expired so that the overlayfs is "flushed" for lower memory consumption?

[yutayu]

@Heiko-L My recognization is resync is just only sync , not for freeing memory.

[manorit2001]

Maybe I don't understand it, but shouldn't everything be copied from the upper to the lower dir after the timer expired so that the overlayfs is "flushed" for lower memory consumption?

Ideally, this should be it but while running tests with clearing the upper directory. There seems to be some issues of data losses due to the internal caching mechanism of overlayFS. It doesn't update the content of the lower directory deterministically as the logic dictates that the lower directory is supposed to be read-only as per overlayFS. So if we try to copy all the content to lower directory and free the upper directory, there seems to be some inconsistencies in the directory mounted with overlayFS. I hope I come across some solution by which I'll be able to implement it but as of now am afraid i can't do much.

[mox-mox]

Hi, I believe this data loss has hit me. To be safe(er) for the future, should/could I disable the overlayfs mode?

[manorit2001]

No, I haven't implemented this so the data loss must not be due to this. The data loss can always occur while using ASD depending on what kind of data loss you're talking about. Think of ASD as a volatile memory with a timeout for syncing back stuff ( by default it is set to one hour ). Suppose you were working on something for the period of one hour and some power cut off occurred before the sync back ( not a proper shutdown ). Now if this occurs the data that was in the volatile memory wouldn't sync back to the disk. However, asd tried to keep the last synced backup intact and is focused on that part. I had created a lot of checks to try to minimize this data loss and all to keep the data intact from the last backup. Have you lost all your data or is it just the volatile data that am talking about? Also, are you on the latest version of ASD or some older version?

[mox-mox]

My current version of ASD is 6.0.0-1. The data loss might have ocured with the previous version, I'm not sure when the Archlinux repo was last updated. My setup involves one folder (scratch) where I do most of my development stuff. In the past few years I've had several unclean shutdowns (partly Out of Memory Exceptions) and have learned to recover my data (minus the last hour of course). A few months back, I've decided to try the overlayfs feature, so I uncommented the respective line in /etc/asd.conf to yield

WHATTOSYNC=('/home/mox/scratch' '/home/mox/projects/hardware/dodgy') # The 'dodgy' folder is irrelevant here because I have not touched it for quite a long time.
USE_OVERLAYFS="yes"

Everything else was commented out. Over the last week(s), I've been working on some project. I did not turn off my computer but I have suspended it serval times. Then, I did a system update (including new kernel if that is relevant) and restarted the computer. When I looked into the scratch folder, the data there was older. At least a weeks worth of code was gone. So naturally, I've looked for a crash recovery snapshot. There were two snapshots (.scratch-backup_asd-crashrecovery-20220626_015926.tar.zstd and .scratch-backup_asd-crashrecovery-20220726_105434.tar.zstd). The older snapshot is too old to be relevant, so I've looked into the newer one. The code in there is quite a bit older than what I've had before the restart.

Next, I've looked into the journal to see if asd has reported any errors. It seems to me it did but I'm not sure how to interpret them. The cropped output of journalctl --no-pager -u asd: (full output was much too long for a comment)

First part where an error was shown

Jun 23 19:36:26 deep-thought systemd[1]: Starting Anything-sync-daemon...
Jun 23 19:36:26 deep-thought anything-sync-daemon[596]: checking flock
Jun 23 19:36:26 deep-thought anything-sync-daemon[596]: flock found
Jun 23 19:36:26 deep-thought anything-sync-daemon[596]: Waiting for lock...
Jun 23 19:36:27 deep-thought anything-sync-daemon[599]: checking flock
Jun 23 19:36:27 deep-thought anything-sync-daemon[599]: flock found
Jun 23 19:36:27 deep-thought anything-sync-daemon[599]: ASDNAME: asd
Jun 23 19:36:27 deep-thought anything-sync-daemon[599]: Configuration file: /etc/asd.conf
Jun 23 19:36:27 deep-thought anything-sync-daemon[599]: Daemon file: /run/asd
Jun 23 19:36:27 deep-thought anything-sync-daemon[599]: Lockfile: /run/asd-lock
Jun 23 19:36:27 deep-thought anything-sync-daemon[599]: Checking for existence of /etc/asd.conf
Jun 23 19:36:27 deep-thought anything-sync-daemon[599]: Checking if asd is already running
Jun 23 19:36:27 deep-thought anything-sync-daemon[599]: Daemon file not found: asd is not running
Jun 23 19:36:27 deep-thought anything-sync-daemon[599]: Backup limit: 5
Jun 23 19:36:27 deep-thought anything-sync-daemon[599]: Volatile dir: /tmp
Jun 23 19:36:27 deep-thought anything-sync-daemon[599]: Hardlink safety check: 1
Jun 23 19:36:27 deep-thought anything-sync-daemon[599]: Backups enabled: yes
Jun 23 19:36:27 deep-thought anything-sync-daemon[599]: Crash recovery enabled: 1
Jun 23 19:36:27 deep-thought anything-sync-daemon[599]: overlayfs enabled: 1
Jun 23 19:36:27 deep-thought anything-sync-daemon[599]: overlayfs version: 23
Jun 23 19:36:27 deep-thought anything-sync-daemon[599]: Checking root permissions
Jun 23 19:36:27 deep-thought anything-sync-daemon[599]: Have root permissions
Jun 23 19:36:27 deep-thought anything-sync-daemon[599]: Checking dependencies
Jun 23 19:36:27 deep-thought anything-sync-daemon[599]: checking rsync
Jun 23 19:36:27 deep-thought anything-sync-daemon[599]: checking awk
Jun 23 19:36:27 deep-thought anything-sync-daemon[599]: checking pv
Jun 23 19:36:27 deep-thought anything-sync-daemon[599]: checking tar
Jun 23 19:36:27 deep-thought anything-sync-daemon[599]: checking zstd
Jun 23 19:36:27 deep-thought anything-sync-daemon[599]: Checking configs
Jun 23 19:36:27 deep-thought anything-sync-daemon[599]: DIR: /home/mox/scratch
Jun 23 19:36:27 deep-thought anything-sync-daemon[599]: BACKUP: /home/mox/.scratch-backup_asd
Jun 23 19:36:27 deep-thought anything-sync-daemon[599]: BACK_OLD: /home/mox/.scratch-backup_asd-old
Jun 23 19:36:28 deep-thought anything-sync-daemon[599]: No hardlinks found
Jun 23 19:36:28 deep-thought anything-sync-daemon[599]: DIR: /home/mox/projects/hardware/dodgy
Jun 23 19:36:28 deep-thought anything-sync-daemon[599]: BACKUP: /home/mox/projects/hardware/.dodgy-backup_asd
Jun 23 19:36:28 deep-thought anything-sync-daemon[599]: BACK_OLD: /home/mox/projects/hardware/.dodgy-backup_asd-old
Jun 23 19:36:28 deep-thought anything-sync-daemon[599]: No hardlinks found
Jun 23 19:36:28 deep-thought anything-sync-daemon[599]: Configs seem to be fine
Jun 23 19:36:28 deep-thought anything-sync-daemon[599]: checking ungraceful state
Jun 23 19:36:28 deep-thought anything-sync-daemon[599]: DIR: /home/mox/scratch
Jun 23 19:36:28 deep-thought anything-sync-daemon[599]: BACKUP: /home/mox/.scratch-backup_asd
Jun 23 19:36:28 deep-thought anything-sync-daemon[599]: BACK_OLD: /home/mox/.scratch-backup_asd-old
Jun 23 19:36:28 deep-thought anything-sync-daemon[599]: USER: mox
Jun 23 19:36:28 deep-thought anything-sync-daemon[599]: TMP: /tmp/asd-mox/home/mox/scratch
Jun 23 19:36:28 deep-thought anything-sync-daemon[599]: No ungraceful state detected
Jun 23 19:36:28 deep-thought anything-sync-daemon[599]: DIR: /home/mox/projects/hardware/dodgy
Jun 23 19:36:28 deep-thought anything-sync-daemon[599]: BACKUP: /home/mox/projects/hardware/.dodgy-backup_asd
Jun 23 19:36:28 deep-thought anything-sync-daemon[599]: BACK_OLD: /home/mox/projects/hardware/.dodgy-backup_asd-old
Jun 23 19:36:28 deep-thought anything-sync-daemon[599]: USER: mox
Jun 23 19:36:28 deep-thought anything-sync-daemon[599]: TMP: /tmp/asd-mox/home/mox/projects/hardware/dodgy
Jun 23 19:36:28 deep-thought anything-sync-daemon[599]: No ungraceful state detected
Jun 23 19:36:28 deep-thought anything-sync-daemon[599]: Syncing files
Jun 23 19:36:28 deep-thought anything-sync-daemon[599]: copied /etc/asd.conf to /run/asd.conf
Jun 23 19:36:28 deep-thought anything-sync-daemon[599]: DIR: /home/mox/scratch
Jun 23 19:36:28 deep-thought anything-sync-daemon[599]: USER: mox
Jun 23 19:36:28 deep-thought anything-sync-daemon[599]: GROUP: 1000
Jun 23 19:36:28 deep-thought anything-sync-daemon[599]: TMP: /tmp/asd-mox/home/mox/scratch
Jun 23 19:36:28 deep-thought anything-sync-daemon[599]: UPPER: /tmp/asd-mox/home/mox/scratch-rw
Jun 23 19:36:28 deep-thought anything-sync-daemon[599]: WORK: /tmp/.asd-mox/home/mox/scratch
Jun 23 19:36:28 deep-thought anything-sync-daemon[599]: ensuring overlay directories
Jun 23 19:36:28 deep-thought anything-sync-daemon[599]: Bind mounting /home/mox/scratch -> /home/mox/.scratch-backup_asd
Jun 23 19:36:28 deep-thought anything-sync-daemon[599]: Creating new linked backup directory /home/mox/.scratch-backup_asd-old
Jun 23 19:36:28 deep-thought anything-sync-daemon[599]: + find /home/mox/.scratch-backup_asd -type l -printf '%P\n'
Jun 23 19:36:28 deep-thought anything-sync-daemon[599]: + rm -rf /home/mox/.scratch-backup_asd-old
Jun 23 19:36:28 deep-thought anything-sync-daemon[599]: + rsync -aX --no-links --link-dest=/home/mox/scratch /home/mox/scratch/ /home/mox/.scratch-backup_asd-old/ --info=progress2
Jun 23 19:36:28 deep-thought anything-sync-daemon[954]: [47.9K blob data]
Jun 23 19:36:29 deep-thought anything-sync-daemon[954]: [48.0K blob data]
Jun 23 19:36:29 deep-thought anything-sync-daemon[954]: [47.9K blob data]
Jun 23 19:36:30 deep-thought anything-sync-daemon[954]: [37.8K blob data]
Jun 23 19:36:30 deep-thought anything-sync-daemon[599]: + rsync -aXl --files-from=/tmp/tmp.JVtTKO3DRE /home/mox/scratch/ /home/mox/.scratch-backup_asd-old/ --info=progress2
Jun 23 19:36:30 deep-thought anything-sync-daemon[1186]: [192B blob data]
Jun 23 19:36:30 deep-thought anything-sync-daemon[599]: + set +x
Jun 23 19:36:30 deep-thought anything-sync-daemon[599]: Mounting overlay directory
Jun 23 19:36:30 deep-thought anything-sync-daemon[599]: bind mounting /tmp/asd-mox/home/mox/scratch -> /home/mox/scratch
Jun 23 19:36:30 deep-thought anything-sync-daemon[599]: DIR: /home/mox/projects/hardware/dodgy
Jun 23 19:36:30 deep-thought anything-sync-daemon[599]: USER: mox
Jun 23 19:36:30 deep-thought anything-sync-daemon[599]: GROUP: 1000
Jun 23 19:36:30 deep-thought anything-sync-daemon[599]: TMP: /tmp/asd-mox/home/mox/projects/hardware/dodgy
Jun 23 19:36:30 deep-thought anything-sync-daemon[599]: UPPER: /tmp/asd-mox/home/mox/projects/hardware/dodgy-rw
Jun 23 19:36:30 deep-thought anything-sync-daemon[599]: WORK: /tmp/.asd-mox/home/mox/projects/hardware/dodgy
Jun 23 19:36:30 deep-thought anything-sync-daemon[599]: ensuring overlay directories
Jun 23 19:36:30 deep-thought anything-sync-daemon[599]: Bind mounting /home/mox/projects/hardware/dodgy -> /home/mox/projects/hardware/.dodgy-backup_asd
Jun 23 19:36:30 deep-thought anything-sync-daemon[599]: Creating new linked backup directory /home/mox/projects/hardware/.dodgy-backup_asd-old
Jun 23 19:36:30 deep-thought anything-sync-daemon[599]: + find /home/mox/projects/hardware/.dodgy-backup_asd -type l -printf '%P\n'
Jun 23 19:36:30 deep-thought anything-sync-daemon[599]: + rm -rf /home/mox/projects/hardware/.dodgy-backup_asd-old
Jun 23 19:36:30 deep-thought anything-sync-daemon[599]: + rsync -aX --no-links --link-dest=/home/mox/projects/hardware/dodgy /home/mox/projects/hardware/dodgy/ /home/mox/projects/hardware/.dodgy-backup_asd-old/ --info=progress2
Jun 23 19:36:30 deep-thought anything-sync-daemon[1204]: skipping non-regular file "literature/tests/build/libs/x86_64/cocotb.vpi"
Jun 23 19:36:30 deep-thought anything-sync-daemon[1204]: skipping non-regular file "literature/tests/build/libs/x86_64/gpivpi.vpl"
Jun 23 19:36:30 deep-thought anything-sync-daemon[1204]: skipping non-regular file "literature/tests/build/libs/x86_64/simulator.so"
Jun 23 19:36:30 deep-thought anything-sync-daemon[1204]: [7.5K blob data]
Jun 23 19:36:30 deep-thought anything-sync-daemon[599]: + rsync -aXl --files-from=/tmp/tmp.QH7nkyS1SN /home/mox/projects/hardware/dodgy/ /home/mox/projects/hardware/.dodgy-backup_asd-old/ --info=progress2
Jun 23 19:36:30 deep-thought anything-sync-daemon[1207]: [192B blob data]
Jun 23 19:36:30 deep-thought anything-sync-daemon[599]: + set +x
Jun 23 19:36:30 deep-thought anything-sync-daemon[599]: Mounting overlay directory
Jun 23 19:36:30 deep-thought anything-sync-daemon[599]: bind mounting /tmp/asd-mox/home/mox/projects/hardware/dodgy -> /home/mox/projects/hardware/dodgy
Jun 23 19:36:30 deep-thought anything-sync-daemon[599]: Sync successful
Jun 23 19:36:30 deep-thought anything-sync-daemon[599]: creating /run/asd
Jun 23 19:36:30 deep-thought anything-sync-daemon[599]: Enforcing number of backups
Jun 23 19:36:30 deep-thought anything-sync-daemon[599]: DIR: /home/mox/scratch
Jun 23 19:36:30 deep-thought anything-sync-daemon[599]: BACKUP: /home/mox/.scratch-backup_asd
Jun 23 19:36:30 deep-thought anything-sync-daemon[599]: The backups are less than 5, nothing to do
Jun 23 19:36:30 deep-thought anything-sync-daemon[599]: DIR: /home/mox/projects/hardware/dodgy
Jun 23 19:36:30 deep-thought anything-sync-daemon[599]: BACKUP: /home/mox/projects/hardware/.dodgy-backup_asd
Jun 23 19:36:30 deep-thought anything-sync-daemon[599]: The backups are less than 5, nothing to do
Jun 23 19:36:30 deep-thought anything-sync-daemon[596]: flock: getting lock took 0.000004 seconds
Jun 23 19:36:30 deep-thought anything-sync-daemon[596]: flock: executing /usr/bin/anything-sync-daemon
Jun 23 19:36:30 deep-thought systemd[1]: Finished Anything-sync-daemon.
Jun 26 01:57:42 deep-thought systemd[1]: Stopping Anything-sync-daemon...
Jun 26 01:57:42 deep-thought anything-sync-daemon[22455]: checking flock
Jun 26 01:57:42 deep-thought anything-sync-daemon[22455]: flock found
Jun 26 01:57:42 deep-thought anything-sync-daemon[22455]: Waiting for lock...
Jun 26 01:57:42 deep-thought anything-sync-daemon[22461]: checking flock
Jun 26 01:57:42 deep-thought anything-sync-daemon[22461]: flock found
Jun 26 01:57:42 deep-thought anything-sync-daemon[22461]: ASDNAME: asd
Jun 26 01:57:42 deep-thought anything-sync-daemon[22461]: Configuration file: /etc/asd.conf
Jun 26 01:57:42 deep-thought anything-sync-daemon[22461]: Daemon file: /run/asd
Jun 26 01:57:42 deep-thought anything-sync-daemon[22461]: Lockfile: /run/asd-lock
Jun 26 01:57:42 deep-thought anything-sync-daemon[22461]: Checking for existence of /etc/asd.conf
Jun 26 01:57:42 deep-thought anything-sync-daemon[22461]: Checking if asd is already running
Jun 26 01:57:42 deep-thought anything-sync-daemon[22461]: Daemon file found: asd is running
Jun 26 01:57:42 deep-thought anything-sync-daemon[22461]: Restoring old asd.conf configurations
Jun 26 01:57:42 deep-thought anything-sync-daemon[22461]: Backup limit: 5
Jun 26 01:57:43 deep-thought anything-sync-daemon[22461]: Volatile dir: /tmp
Jun 26 01:57:43 deep-thought anything-sync-daemon[22461]: Hardlink safety check: 1
Jun 26 01:57:43 deep-thought anything-sync-daemon[22461]: Backups enabled: yes
Jun 26 01:57:43 deep-thought anything-sync-daemon[22461]: Crash recovery enabled: 1
Jun 26 01:57:43 deep-thought anything-sync-daemon[22461]: overlayfs enabled: 1
Jun 26 01:57:43 deep-thought anything-sync-daemon[22461]: overlayfs version: 23
Jun 26 01:57:43 deep-thought anything-sync-daemon[22461]: Checking root permissions
Jun 26 01:57:43 deep-thought anything-sync-daemon[22461]: Have root permissions
Jun 26 01:57:43 deep-thought anything-sync-daemon[22461]: checking ungraceful state
Jun 26 01:57:43 deep-thought anything-sync-daemon[22461]: DIR: /home/mox/scratch
Jun 26 01:57:43 deep-thought anything-sync-daemon[22461]: BACKUP: /home/mox/.scratch-backup_asd
Jun 26 01:57:43 deep-thought anything-sync-daemon[22461]: BACK_OLD: /home/mox/.scratch-backup_asd-old
Jun 26 01:57:43 deep-thought anything-sync-daemon[22461]: USER: mox
Jun 26 01:57:43 deep-thought anything-sync-daemon[22461]: TMP: /tmp/asd-mox/home/mox/scratch
Jun 26 01:57:43 deep-thought anything-sync-daemon[22461]: No ungraceful state detected
Jun 26 01:57:43 deep-thought anything-sync-daemon[22461]: DIR: /home/mox/projects/hardware/dodgy
Jun 26 01:57:43 deep-thought anything-sync-daemon[22461]: BACKUP: /home/mox/projects/hardware/.dodgy-backup_asd
Jun 26 01:57:43 deep-thought anything-sync-daemon[22461]: BACK_OLD: /home/mox/projects/hardware/.dodgy-backup_asd-old
Jun 26 01:57:43 deep-thought anything-sync-daemon[22461]: USER: mox
Jun 26 01:57:43 deep-thought anything-sync-daemon[22461]: TMP: /tmp/asd-mox/home/mox/projects/hardware/dodgy
Jun 26 01:57:43 deep-thought anything-sync-daemon[22461]: No ungraceful state detected
Jun 26 01:57:43 deep-thought anything-sync-daemon[22461]: Syncing files
Jun 26 01:57:43 deep-thought anything-sync-daemon[22461]: DIR: /home/mox/scratch
Jun 26 01:57:43 deep-thought anything-sync-daemon[22461]: USER: mox
Jun 26 01:57:43 deep-thought anything-sync-daemon[22461]: GROUP: 1000
Jun 26 01:57:43 deep-thought anything-sync-daemon[22461]: TMP: /tmp/asd-mox/home/mox/scratch
Jun 26 01:57:43 deep-thought anything-sync-daemon[22461]: UPPER: /tmp/asd-mox/home/mox/scratch-rw
Jun 26 01:57:43 deep-thought anything-sync-daemon[22461]: WORK: /tmp/.asd-mox/home/mox/scratch
Jun 26 01:57:43 deep-thought anything-sync-daemon[22461]: ensuring overlay directories
Jun 26 01:57:43 deep-thought anything-sync-daemon[22461]: Syncing /tmp/asd-mox/home/mox/scratch and /home/mox/.scratch-backup_asd
Jun 26 01:57:43 deep-thought anything-sync-daemon[22461]: + rsync -aX --delete-after --exclude .flagged /tmp/asd-mox/home/mox/scratch/ /home/mox/.scratch-backup_asd/ --info=progress2
Jun 26 01:57:43 deep-thought anything-sync-daemon[22492]: rsync: [sender] readdir("/tmp/asd-mox/home/mox/scratch/testpilot/target/debug/incremental/testpilot-2cqkmcm7lefjo/s-gb2ocnkrh9-xxj1wh-3r9a0nmn0emey"): Input/output error (5)
Jun 26 01:57:43 deep-thought anything-sync-daemon[22492]: [978B blob data]
Jun 26 01:57:43 deep-thought anything-sync-daemon[22492]: IO error encountered -- skipping file deletion
Jun 26 01:57:44 deep-thought anything-sync-daemon[22492]: rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1326) [sender=v3.2.4]
Jun 26 01:57:44 deep-thought anything-sync-daemon[22455]: flock: getting lock took 0.000003 seconds
Jun 26 01:57:44 deep-thought anything-sync-daemon[22455]: flock: executing /usr/bin/anything-sync-daemon
Jun 26 01:57:44 deep-thought systemd[1]: asd.service: Control process exited, code=exited, status=23/n/a
Jun 26 01:57:44 deep-thought systemd[1]: asd.service: Failed with result 'exit-code'.
Jun 26 01:57:44 deep-thought systemd[1]: Stopped Anything-sync-daemon.
Jun 26 01:57:44 deep-thought systemd[1]: asd.service: Consumed 2.819s CPU time.
-- Boot f82790b158ab479d967c18823f23c590 --

and

Second part where an error was thrown

-- Boot c9c8b9cde0794af1a836df86a7638f7f --
Jul 18 13:03:48 deep-thought systemd[1]: Starting Anything-sync-daemon...
Jul 18 13:03:48 deep-thought anything-sync-daemon[566]: checking flock
Jul 18 13:03:48 deep-thought anything-sync-daemon[566]: flock found
Jul 18 13:03:48 deep-thought anything-sync-daemon[566]: Waiting for lock...
Jul 18 13:03:48 deep-thought anything-sync-daemon[571]: checking flock
Jul 18 13:03:48 deep-thought anything-sync-daemon[571]: flock found
Jul 18 13:03:48 deep-thought anything-sync-daemon[571]: ASDNAME: asd
Jul 18 13:03:48 deep-thought anything-sync-daemon[571]: Configuration file: /etc/asd.conf
Jul 18 13:03:48 deep-thought anything-sync-daemon[571]: Daemon file: /run/asd
Jul 18 13:03:48 deep-thought anything-sync-daemon[571]: Lockfile: /run/asd-lock
Jul 18 13:03:48 deep-thought anything-sync-daemon[571]: Checking for existence of /etc/asd.conf
Jul 18 13:03:48 deep-thought anything-sync-daemon[571]: Checking if asd is already running
Jul 18 13:03:48 deep-thought anything-sync-daemon[571]: Daemon file not found: asd is not running
Jul 18 13:03:48 deep-thought anything-sync-daemon[571]: Backup limit: 5
Jul 18 13:03:48 deep-thought anything-sync-daemon[571]: Volatile dir: /tmp
Jul 18 13:03:48 deep-thought anything-sync-daemon[571]: Hardlink safety check: 1
Jul 18 13:03:48 deep-thought anything-sync-daemon[571]: Backups enabled: yes
Jul 18 13:03:48 deep-thought anything-sync-daemon[571]: Crash recovery enabled: 1
Jul 18 13:03:48 deep-thought anything-sync-daemon[571]: overlayfs enabled: 1
Jul 18 13:03:48 deep-thought anything-sync-daemon[571]: overlayfs version: 23
Jul 18 13:03:48 deep-thought anything-sync-daemon[571]: Checking root permissions
Jul 18 13:03:48 deep-thought anything-sync-daemon[571]: Have root permissions
Jul 18 13:03:48 deep-thought anything-sync-daemon[571]: Checking dependencies
Jul 18 13:03:48 deep-thought anything-sync-daemon[571]: checking rsync
Jul 18 13:03:48 deep-thought anything-sync-daemon[571]: checking awk
Jul 18 13:03:48 deep-thought anything-sync-daemon[571]: checking pv
Jul 18 13:03:48 deep-thought anything-sync-daemon[571]: checking tar
Jul 18 13:03:48 deep-thought anything-sync-daemon[571]: checking zstd
Jul 18 13:03:48 deep-thought anything-sync-daemon[571]: Checking configs
Jul 18 13:03:48 deep-thought anything-sync-daemon[571]: DIR: /home/mox/scratch
Jul 18 13:03:48 deep-thought anything-sync-daemon[571]: BACKUP: /home/mox/.scratch-backup_asd
Jul 18 13:03:48 deep-thought anything-sync-daemon[571]: BACK_OLD: /home/mox/.scratch-backup_asd-old
Jul 18 13:03:49 deep-thought anything-sync-daemon[571]: No hardlinks found
Jul 18 13:03:49 deep-thought anything-sync-daemon[571]: DIR: /home/mox/projects/hardware/dodgy
Jul 18 13:03:49 deep-thought anything-sync-daemon[571]: BACKUP: /home/mox/projects/hardware/.dodgy-backup_asd
Jul 18 13:03:49 deep-thought anything-sync-daemon[571]: BACK_OLD: /home/mox/projects/hardware/.dodgy-backup_asd-old
Jul 18 13:03:49 deep-thought anything-sync-daemon[571]: No hardlinks found
Jul 18 13:03:49 deep-thought anything-sync-daemon[571]: Configs seem to be fine
Jul 18 13:03:49 deep-thought anything-sync-daemon[571]: checking ungraceful state
Jul 18 13:03:49 deep-thought anything-sync-daemon[571]: DIR: /home/mox/scratch
Jul 18 13:03:49 deep-thought anything-sync-daemon[571]: BACKUP: /home/mox/.scratch-backup_asd
Jul 18 13:03:49 deep-thought anything-sync-daemon[571]: BACK_OLD: /home/mox/.scratch-backup_asd-old
Jul 18 13:03:49 deep-thought anything-sync-daemon[571]: USER: mox
Jul 18 13:03:49 deep-thought anything-sync-daemon[571]: TMP: /tmp/asd-mox/home/mox/scratch
Jul 18 13:03:49 deep-thought anything-sync-daemon[571]: No ungraceful state detected
Jul 18 13:03:49 deep-thought anything-sync-daemon[571]: DIR: /home/mox/projects/hardware/dodgy
Jul 18 13:03:49 deep-thought anything-sync-daemon[571]: BACKUP: /home/mox/projects/hardware/.dodgy-backup_asd
Jul 18 13:03:49 deep-thought anything-sync-daemon[571]: BACK_OLD: /home/mox/projects/hardware/.dodgy-backup_asd-old
Jul 18 13:03:49 deep-thought anything-sync-daemon[571]: USER: mox
Jul 18 13:03:49 deep-thought anything-sync-daemon[571]: TMP: /tmp/asd-mox/home/mox/projects/hardware/dodgy
Jul 18 13:03:49 deep-thought anything-sync-daemon[571]: No ungraceful state detected
Jul 18 13:03:49 deep-thought anything-sync-daemon[571]: Syncing files
Jul 18 13:03:49 deep-thought anything-sync-daemon[571]: copied /etc/asd.conf to /run/asd.conf
Jul 18 13:03:49 deep-thought anything-sync-daemon[571]: DIR: /home/mox/scratch
Jul 18 13:03:49 deep-thought anything-sync-daemon[571]: USER: mox
Jul 18 13:03:49 deep-thought anything-sync-daemon[571]: GROUP: 1000
Jul 18 13:03:49 deep-thought anything-sync-daemon[571]: TMP: /tmp/asd-mox/home/mox/scratch
Jul 18 13:03:49 deep-thought anything-sync-daemon[571]: UPPER: /tmp/asd-mox/home/mox/scratch-rw
Jul 18 13:03:49 deep-thought anything-sync-daemon[571]: WORK: /tmp/.asd-mox/home/mox/scratch
Jul 18 13:03:49 deep-thought anything-sync-daemon[571]: ensuring overlay directories
Jul 18 13:03:49 deep-thought anything-sync-daemon[571]: Bind mounting /home/mox/scratch -> /home/mox/.scratch-backup_asd
Jul 18 13:03:49 deep-thought anything-sync-daemon[571]: Creating new linked backup directory /home/mox/.scratch-backup_asd-old
Jul 18 13:03:49 deep-thought anything-sync-daemon[571]: + find /home/mox/.scratch-backup_asd -type l -printf '%P\n'
Jul 18 13:03:49 deep-thought anything-sync-daemon[571]: + rm -rf /home/mox/.scratch-backup_asd-old
Jul 18 13:03:49 deep-thought anything-sync-daemon[571]: + rsync -aX --no-links --link-dest=/home/mox/scratch /home/mox/scratch/ /home/mox/.scratch-backup_asd-old/ --info=progress2
Jul 18 13:03:50 deep-thought anything-sync-daemon[906]: [47.9K blob data]
Jul 18 13:03:50 deep-thought anything-sync-daemon[906]: [48.0K blob data]
Jul 18 13:03:51 deep-thought anything-sync-daemon[906]: [47.9K blob data]
Jul 18 13:03:52 deep-thought anything-sync-daemon[906]: [44.5K blob data]
Jul 18 13:03:52 deep-thought anything-sync-daemon[571]: + rsync -aXl --files-from=/tmp/tmp.TB4sD1eyw4 /home/mox/scratch/ /home/mox/.scratch-backup_asd-old/ --info=progress2
Jul 18 13:03:52 deep-thought anything-sync-daemon[1159]: [192B blob data]
Jul 18 13:03:52 deep-thought anything-sync-daemon[571]: + set +x
Jul 18 13:03:52 deep-thought anything-sync-daemon[571]: Mounting overlay directory
Jul 18 13:03:52 deep-thought anything-sync-daemon[571]: bind mounting /tmp/asd-mox/home/mox/scratch -> /home/mox/scratch
Jul 18 13:03:52 deep-thought anything-sync-daemon[571]: DIR: /home/mox/projects/hardware/dodgy
Jul 18 13:03:52 deep-thought anything-sync-daemon[571]: USER: mox
Jul 18 13:03:52 deep-thought anything-sync-daemon[571]: GROUP: 1000
Jul 18 13:03:52 deep-thought anything-sync-daemon[571]: TMP: /tmp/asd-mox/home/mox/projects/hardware/dodgy
Jul 18 13:03:52 deep-thought anything-sync-daemon[571]: UPPER: /tmp/asd-mox/home/mox/projects/hardware/dodgy-rw
Jul 18 13:03:52 deep-thought anything-sync-daemon[571]: WORK: /tmp/.asd-mox/home/mox/projects/hardware/dodgy
Jul 18 13:03:52 deep-thought anything-sync-daemon[571]: ensuring overlay directories
Jul 18 13:03:52 deep-thought anything-sync-daemon[571]: Bind mounting /home/mox/projects/hardware/dodgy -> /home/mox/projects/hardware/.dodgy-backup_asd
Jul 18 13:03:52 deep-thought anything-sync-daemon[571]: Creating new linked backup directory /home/mox/projects/hardware/.dodgy-backup_asd-old
Jul 18 13:03:52 deep-thought anything-sync-daemon[571]: + find /home/mox/projects/hardware/.dodgy-backup_asd -type l -printf '%P\n'
Jul 18 13:03:52 deep-thought anything-sync-daemon[571]: + rm -rf /home/mox/projects/hardware/.dodgy-backup_asd-old
Jul 18 13:03:52 deep-thought anything-sync-daemon[571]: + rsync -aX --no-links --link-dest=/home/mox/projects/hardware/dodgy /home/mox/projects/hardware/dodgy/ /home/mox/projects/hardware/.dodgy-backup_asd-old/ --info=progress2
Jul 18 13:03:52 deep-thought anything-sync-daemon[1177]: skipping non-regular file "literature/tests/build/libs/x86_64/cocotb.vpi"
Jul 18 13:03:52 deep-thought anything-sync-daemon[1177]: skipping non-regular file "literature/tests/build/libs/x86_64/gpivpi.vpl"
Jul 18 13:03:52 deep-thought anything-sync-daemon[1177]: skipping non-regular file "literature/tests/build/libs/x86_64/simulator.so"
Jul 18 13:03:52 deep-thought anything-sync-daemon[1177]: [7.5K blob data]
Jul 18 13:03:52 deep-thought anything-sync-daemon[571]: + rsync -aXl --files-from=/tmp/tmp.VIAShDpMGP /home/mox/projects/hardware/dodgy/ /home/mox/projects/hardware/.dodgy-backup_asd-old/ --info=progress2
Jul 18 13:03:52 deep-thought anything-sync-daemon[1180]: [192B blob data]
Jul 18 13:03:52 deep-thought anything-sync-daemon[571]: + set +x
Jul 18 13:03:52 deep-thought anything-sync-daemon[571]: Mounting overlay directory
Jul 18 13:03:52 deep-thought anything-sync-daemon[571]: bind mounting /tmp/asd-mox/home/mox/projects/hardware/dodgy -> /home/mox/projects/hardware/dodgy
Jul 18 13:03:52 deep-thought anything-sync-daemon[571]: Sync successful
Jul 18 13:03:52 deep-thought anything-sync-daemon[571]: creating /run/asd
Jul 18 13:03:52 deep-thought anything-sync-daemon[571]: Enforcing number of backups
Jul 18 13:03:52 deep-thought anything-sync-daemon[571]: DIR: /home/mox/scratch
Jul 18 13:03:52 deep-thought anything-sync-daemon[571]: BACKUP: /home/mox/.scratch-backup_asd
Jul 18 13:03:52 deep-thought anything-sync-daemon[571]: The backups are less than 5, nothing to do
Jul 18 13:03:52 deep-thought anything-sync-daemon[571]: DIR: /home/mox/projects/hardware/dodgy
Jul 18 13:03:52 deep-thought anything-sync-daemon[571]: BACKUP: /home/mox/projects/hardware/.dodgy-backup_asd
Jul 18 13:03:52 deep-thought anything-sync-daemon[571]: The backups are less than 5, nothing to do
Jul 18 13:03:52 deep-thought anything-sync-daemon[566]: flock: getting lock took 0.000003 seconds
Jul 18 13:03:52 deep-thought anything-sync-daemon[566]: flock: executing /usr/bin/anything-sync-daemon
Jul 18 13:03:52 deep-thought systemd[1]: Finished Anything-sync-daemon.
Jul 20 14:20:00 deep-thought systemd[1]: Stopping Anything-sync-daemon...
Jul 20 14:20:00 deep-thought anything-sync-daemon[115206]: checking flock
Jul 20 14:20:00 deep-thought anything-sync-daemon[115206]: flock found
Jul 20 14:20:00 deep-thought anything-sync-daemon[115206]: Waiting for lock...
Jul 20 14:20:00 deep-thought anything-sync-daemon[115207]: checking flock
Jul 20 14:20:00 deep-thought anything-sync-daemon[115207]: flock found
Jul 20 14:20:00 deep-thought anything-sync-daemon[115207]: ASDNAME: asd
Jul 20 14:20:00 deep-thought anything-sync-daemon[115207]: Configuration file: /etc/asd.conf
Jul 20 14:20:00 deep-thought anything-sync-daemon[115207]: Daemon file: /run/asd
Jul 20 14:20:00 deep-thought anything-sync-daemon[115207]: Lockfile: /run/asd-lock
Jul 20 14:20:00 deep-thought anything-sync-daemon[115207]: Checking for existence of /etc/asd.conf
Jul 20 14:20:00 deep-thought anything-sync-daemon[115207]: Checking if asd is already running
Jul 20 14:20:00 deep-thought anything-sync-daemon[115207]: Daemon file found: asd is running
Jul 20 14:20:00 deep-thought anything-sync-daemon[115207]: Restoring old asd.conf configurations
Jul 20 14:20:00 deep-thought anything-sync-daemon[115207]: Backup limit: 5
Jul 20 14:20:00 deep-thought anything-sync-daemon[115207]: Volatile dir: /tmp
Jul 20 14:20:00 deep-thought anything-sync-daemon[115207]: Hardlink safety check: 1
Jul 20 14:20:00 deep-thought anything-sync-daemon[115207]: Backups enabled: yes
Jul 20 14:20:00 deep-thought anything-sync-daemon[115207]: Crash recovery enabled: 1
Jul 20 14:20:00 deep-thought anything-sync-daemon[115207]: overlayfs enabled: 1
Jul 20 14:20:00 deep-thought anything-sync-daemon[115207]: overlayfs version: 23
Jul 20 14:20:00 deep-thought anything-sync-daemon[115207]: Checking root permissions
Jul 20 14:20:00 deep-thought anything-sync-daemon[115207]: Have root permissions
Jul 20 14:20:00 deep-thought anything-sync-daemon[115207]: checking ungraceful state
Jul 20 14:20:00 deep-thought anything-sync-daemon[115207]: DIR: /home/mox/scratch
Jul 20 14:20:00 deep-thought anything-sync-daemon[115207]: BACKUP: /home/mox/.scratch-backup_asd
Jul 20 14:20:00 deep-thought anything-sync-daemon[115207]: BACK_OLD: /home/mox/.scratch-backup_asd-old
Jul 20 14:20:00 deep-thought anything-sync-daemon[115207]: USER: mox
Jul 20 14:20:00 deep-thought anything-sync-daemon[115207]: TMP: /tmp/asd-mox/home/mox/scratch
Jul 20 14:20:00 deep-thought anything-sync-daemon[115207]: No ungraceful state detected
Jul 20 14:20:00 deep-thought anything-sync-daemon[115207]: DIR: /home/mox/projects/hardware/dodgy
Jul 20 14:20:00 deep-thought anything-sync-daemon[115207]: BACKUP: /home/mox/projects/hardware/.dodgy-backup_asd
Jul 20 14:20:00 deep-thought anything-sync-daemon[115207]: BACK_OLD: /home/mox/projects/hardware/.dodgy-backup_asd-old
Jul 20 14:20:00 deep-thought anything-sync-daemon[115207]: USER: mox
Jul 20 14:20:00 deep-thought anything-sync-daemon[115207]: TMP: /tmp/asd-mox/home/mox/projects/hardware/dodgy
Jul 20 14:20:00 deep-thought anything-sync-daemon[115207]: No ungraceful state detected
Jul 20 14:20:00 deep-thought anything-sync-daemon[115207]: Syncing files
Jul 20 14:20:00 deep-thought anything-sync-daemon[115207]: DIR: /home/mox/scratch
Jul 20 14:20:00 deep-thought anything-sync-daemon[115207]: USER: mox
Jul 20 14:20:00 deep-thought anything-sync-daemon[115207]: GROUP: 1000
Jul 20 14:20:00 deep-thought anything-sync-daemon[115207]: TMP: /tmp/asd-mox/home/mox/scratch
Jul 20 14:20:00 deep-thought anything-sync-daemon[115207]: UPPER: /tmp/asd-mox/home/mox/scratch-rw
Jul 20 14:20:00 deep-thought anything-sync-daemon[115207]: WORK: /tmp/.asd-mox/home/mox/scratch
Jul 20 14:20:00 deep-thought anything-sync-daemon[115207]: ensuring overlay directories
Jul 20 14:20:00 deep-thought anything-sync-daemon[115207]: Syncing /tmp/asd-mox/home/mox/scratch and /home/mox/.scratch-backup_asd
Jul 20 14:20:00 deep-thought anything-sync-daemon[115207]: + rsync -aX --delete-after --exclude .flagged /tmp/asd-mox/home/mox/scratch/ /home/mox/.scratch-backup_asd/ --info=progress2
Jul 20 14:20:00 deep-thought anything-sync-daemon[115218]: rsync: [sender] readdir("/tmp/asd-mox/home/mox/scratch/testpilot/target/debug/incremental/testpilot-nimpvpsjv76r/s-gbtlmlnyfe-1et1mbv-working"): Input/output error (5)
Jul 20 14:20:00 deep-thought anything-sync-daemon[115218]: rsync: [sender] readdir("/tmp/asd-mox/home/mox/scratch/testpilot/target/debug/incremental/testpilot-nimpvpsjv76r/s-gbtmppr9f1-w49234-working"): Input/output error (5)
Jul 20 14:20:00 deep-thought anything-sync-daemon[115218]: rsync: [sender] readdir("/tmp/asd-mox/home/mox/scratch/testpilot/target/debug/incremental/testpilot-nimpvpsjv76r/s-gbtmqw911k-l7dsp1-working"): Input/output error (5)
Jul 20 14:20:00 deep-thought anything-sync-daemon[115218]: rsync: [sender] readdir("/tmp/asd-mox/home/mox/scratch/testpilot/target/debug/incremental/testpilot-nimpvpsjv76r/s-gbtmr56y7r-1gmgrkr-working"): Input/output error (5)
Jul 20 14:20:00 deep-thought anything-sync-daemon[115218]: rsync: [sender] readdir("/tmp/asd-mox/home/mox/scratch/testpilot/target/debug/incremental/testpilot-nimpvpsjv76r/s-gbtmqekpm4-1lzdu8n-working"): Input/output error (5)
Jul 20 14:20:00 deep-thought anything-sync-daemon[115218]: rsync: [sender] readdir("/tmp/asd-mox/home/mox/scratch/testpilot/target/debug/incremental/testpilot-nimpvpsjv76r/s-gbtmpfpifg-1h3zbra-working"): Input/output error (5)
Jul 20 14:20:00 deep-thought anything-sync-daemon[115218]: rsync: [sender] readdir("/tmp/asd-mox/home/mox/scratch/testpilot/target/debug/incremental/testpilot-nimpvpsjv76r/s-gbtms4xrvj-1dyjrjd-3a4qm1t6bimyf"): Input/output error (5)
Jul 20 14:20:00 deep-thought anything-sync-daemon[115218]: rsync: [sender] readdir("/tmp/asd-mox/home/mox/scratch/testpilot/target/debug/incremental/testpilot-nimpvpsjv76r/s-gbsrmm8fd8-jy5y2-working"): Input/output error (5)
Jul 20 14:20:00 deep-thought anything-sync-daemon[115218]: [395B blob data]
Jul 20 14:20:00 deep-thought anything-sync-daemon[115218]: IO error encountered -- skipping file deletion
Jul 20 14:20:00 deep-thought anything-sync-daemon[115218]: rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1326) [sender=v3.2.4]
Jul 20 14:20:00 deep-thought anything-sync-daemon[115206]: flock: getting lock took 0.000003 seconds
Jul 20 14:20:00 deep-thought anything-sync-daemon[115206]: flock: executing /usr/bin/anything-sync-daemon
Jul 20 14:20:00 deep-thought systemd[1]: asd.service: Control process exited, code=exited, status=23/n/a
Jul 20 14:20:00 deep-thought systemd[1]: asd.service: Failed with result 'exit-code'.
Jul 20 14:20:00 deep-thought systemd[1]: Stopped Anything-sync-daemon.
Jul 20 14:20:00 deep-thought systemd[1]: asd.service: Consumed 2.468s CPU time.
-- Boot b50f8fc183ba4e6789c86ca70b79801e --

Thank you for taking time to respond and look into this. ASD is a wonderfull piece of software :)

[manorit2001]

I think I can guess what happened, thought I don't know why the I/O errors are occurring in rsync, that usually doesn't occur or I haven't seen it happen. Mostly faulty harddrives might cause that as per few google searches on it. Though regarding the data loss, I think I maybe have a hint on what might've happened. The new version of asd contains a major change which consist of changing the symlinking method of syncing to bind mounting which changed a lot of stuff in the codebase ( hence version upgrade to 6.0 ). Now since you updated the system, your initial setup was done through the previous version but once upgraded, the de-sync happened with the newer version which assumed that bind mounts was used as the setup mechanism. Though it was fault on my part that I didn't know asd install script in the arch repository had some hardcoded offsets for the version changes in which asd would stop ( i recently became a maintainer of this repository ).

$ cat anything-sync-daemon.install                                                                                                               
[...]
pre_upgrade() {
  for version in 5.69 5.76; do
    if [ "$(vercmp "$2" "$version")" -lt 0 ]; then
      _stop_asd_service
      echo 'Start the asd service manually after upgrade is complete.'
    fi
  done
}
[...]

Though once I got to know about it I had updated @graysky2 about this change and mentioned him to update the arch scripts to stop asd before any upgrade ( regardless of the version numbers ) though I can still see the old scripts only available in the repo.

[mox-mox]

Wow. That's quite a comprehensive analysis. If I understand this correctly, ignoring the I/O errors my data loss was mainly due to the (more or less faulty) update process in the Arch Linux package and this will soon be fixed. Thank you very much! :)