I just lost 200TB of data

[Motophan]

I have

/mnt/btrfs01 /mnt/btrfs02 /mnt/btrfs03 /mnt/zfs01 /mnt/zfs02

merged together to /mnt/mergerfs

/mnt/btrfs01, /mnt/btrfs02, /mnt/btrfs03 are 90% full /mnt/zfs01, /mnt/zfs02 are 0% full

I setup mergerfs policy for ff (new files are added to first listed)

[Unit] Description=mergerfs service

[Service]
Type=simple
KillMode=none
ExecStart=/usr/bin/mergerfs \
  -f \
  -o cache.files=partial \
  -o allow_other \
  -o category.create=ff \
  -o minfreespace=300G \
  -o use_ino \
  -o dropcacheonclose=true \
  -o ignorepponrename=true \
  -o moveonenospc=mspmfs \
  -o func.getattr=newest \
  /mnt/zfs01:/mnt/zfs02:/mnt/btrfs01:/mnt/btrfs02:/mnt/btrfs03 \
  /mnt/mergerfs
ExecStop=/bin/fusermount -uz /mnt/mergerfs
Restart=on-failure

[Install]
WantedBy=default.target

So new files added to /mnt/mergerfs should go into /mnt/zfs01, then once /mnt/zfs01 has less than 300GB free space it should go into /mnt/zfs02 and so on.

rsync -av --progress --remove-source-files --exclude=".*" --bwlimit=300M /mnt/btrfs01/ /mnt/mergerfs/

/mnt/btrfs0X/.btrfs/ is where snapshots are so it wasnt dataloss issue

this deleted 200+TB of data and didnt put it on /mnt/zfs01

:(

[bland328]

I've never used mergerfs, so this is a shot in the dark, but does this configuration mean that files which are actually stored in /mnt/btrfs01 also appear as virtual representations under /mnt/mergerfs?

If I'm right about that, then perhaps the problem is that each file found to already exist at the "destination" is therefore deleted from the "source," the problem being that both source and destination files are secretly the same file.

Again, just a guess, but I hope it might help.

[Motophan]

thats exactly what happened

w/ no warning the program was like "yolo lets just delete 200TB of files :) "

[akostadinov]

@Motophan , so how would rsync detect that? Is this issue actionable?

[mfwitten]

You told rsync to perform these steps:

0. Synchronize files in /mnt/mergerfs with files in /mnt/btrfs01/.
1. After that synchronization, delete from /mnt/btrfs01 all the files that were synchronized.

So, rsync did exactly what it was told to do:

0. It found the set of files, F, to synchronize.
1. It made sure they were indeed synchronized; it made sure those files existed in /mnt/mergerfs.
2. It deleted all the files in F from /mnt/btrfs01, because they are no longer needed there; of course, this effectively removed those files from /mnt/mergerfs as well.

Reflect heavily on the origin of the mistake; the part of the program that had a bug in it seems to be the part that exists between the keyboard and chair. Hey. We've all been there!