snazzer
snazzer copied to clipboard
csirac2
failed test for snazzer-send-wrapper
When executing the new make test, the following test fails:
✗ sudo -n snazzer --list-snapshots '--all' 'foo=" some stuff "' 'hel'\\'' squot '\\''lo' 'asd " dquot " fgh' 'ap ple' ' bon'\\''squot'\\''jour' 'there'
(in test file tests/snazzer-send-wrapper.bats, line 45)
`[ "$status" = "0" ]' failed
I'm not sure what the problem is there, though.
Related: Did you try to run all tests and not only snazzer-prune-candidates via travis, maybe the beta image with Ubuntu 14.04 contains a new-enough btrfs for the tests to work?
Unfortunately, sstephenson/bats is great for running tests but not debugging them. My strategy originally was to add echo "$output" >>/tmp/something to the testcases. Perhaps we should switch to one of the better maintained forks of bats. Could you retry the tests with https://github.com/harschware/bats ?
PATH=.:$PATH bats -r -o /tmp/log tests/ seems to emit some useful stdout, and the it seems some stderr might be going to /tmp/log in my quick tests of the harschware fork.
Regarding travis, the real reason it isn't running the bats tests is because it requires root privileges to create and mount manipulate filesystems. I'm quite sure the containers in travis don't permit this sort of thing, at least they didn't last time I tried, admittedly a long time ago now... happy to be proven wrong. If I forked snazzer into a community group I help run then I could get a CI build VM running over there if that's interesting.
Managed to get travis to work, see #34, hurray. Can't reproduce the test failure there, though.
On the contrary, harschware/bats didn't work at all, it only produces the following output and then stalls forever for some reason:
[florian@izumo snazzer]$ PATH=.:$PATH bats -r -o /tmp/log tests/
[sudo] Passwort für florian:
usage: btrfs [--help] [--version] <group> [<group>...] <command> [<args>]
btrfs subvolume create [-i <qgroupid>] [<dest>/]<name>
Create a subvolume
btrfs subvolume delete [options] <subvolume> [<subvolume>...]
Delete subvolume(s)
btrfs subvolume list [options] [-G [+|-]value] [-C [+|-]value] [--sort=gen,ogen,rootid,path] <path>
List subvolumes (and snapshots)
btrfs subvolume snapshot [-r] [-i <qgroupid>] <source> <dest>|[<dest>/]<name>
Create a snapshot of the subvolume
btrfs subvolume get-default <path>
Get the default subvolume of a filesystem
btrfs subvolume set-default <subvolid> <path>
Set the default subvolume of a filesystem
btrfs subvolume find-new <path> <lastgen>
List the recently modified files in a filesystem
btrfs subvolume show <subvol-path>
Show more information of the subvolume
btrfs subvolume sync <path> [<subvol-id>...]
Wait until given subvolume(s) are completely removed from the filesystem.
btrfs filesystem df [options] <path>
Show space usage information for a mount point
btrfs filesystem du [options] <path> [<path>..]
Summarize disk usage of each file.
btrfs filesystem show [options] [<path>|<uuid>|<device>|label]
Show the structure of a filesystem
btrfs filesystem sync <path>
Force a sync on a filesystem
btrfs filesystem defragment [options] <file>|<dir> [<file>|<dir>...]
Defragment a file or a directory
btrfs filesystem resize [devid:][+/-]<newsize>[kKmMgGtTpPeE]|[devid:]max <path>
Resize a filesystem
btrfs filesystem label [<device>|<mount_point>] [<newlabel>]
Get or change the label of a filesystem
btrfs filesystem usage [options] <path> [<path>..]
Show detailed information about internal filesystem usage .
btrfs balance start [options] <path>
Balance chunks across the devices
btrfs balance pause <path>
Pause running balance
btrfs balance cancel <path>
Cancel running or paused balance
btrfs balance resume <path>
Resume interrupted balance
btrfs balance status [-v] <path>
Show status of running or paused balance
btrfs device add [options] <device> [<device>...] <path>
Add a device to a filesystem
btrfs device delete <device>|<devid> [<device>|<devid>...] <path> btrfs device remove <device>|<devid> [<device>|<devid>...] <path>
Remove a device from a filesystem
btrfs device scan [(-d|--all-devices)|<device> [<device>...]]
Scan devices for a btrfs filesystem
btrfs device ready <device>
Check device to see if it has all of its devices in cache for mounting
btrfs device stats [-z] <path>|<device>
Show current device IO stats.
btrfs device usage [options] <path> [<path>..]
Show detailed information about internal allocations in devices.
btrfs scrub start [-BdqrRf] [-c ioprio_class -n ioprio_classdata] <path>|<device>
Start a new scrub. If a scrub is already running, the new one fails.
btrfs scrub cancel <path>|<device>
Cancel a running scrub
btrfs scrub resume [-BdqrR] [-c ioprio_class -n ioprio_classdata] <path>|<device>
Resume previously canceled or interrupted scrub
btrfs scrub status [-dR] <path>|<device>
Show status of running or finished scrub
btrfs check [options] <device>
Check structural integrity of a filesystem (unmounted).
btrfs rescue chunk-recover [options] <device>
Recover the chunk tree by scanning the devices one by one.
btrfs rescue super-recover [options] <device>
Recover bad superblocks from good copies
btrfs rescue zero-log <device>
Clear the tree log. Usable if it's corrupted and prevents mount.
btrfs restore [options] <device> <path> | -l <device>
Try to restore files from a damaged filesystem (unmounted)
btrfs inspect-internal inode-resolve [-v] <inode> <path>
Get file system paths for the given inode
btrfs inspect-internal logical-resolve [-Pv] [-s bufsize] <logical> <path>
Get file system paths for the given logical address
btrfs inspect-internal subvolid-resolve <subvolid> <path>
Get file system paths for the given subvolume ID.
btrfs inspect-internal rootid <path>
Get tree ID of the containing subvolume of path.
btrfs inspect-internal min-dev-size [options] <path>
Get the minimum size the device can be shrunk to. The
btrfs inspect-internal dump-tree [options] device
Dump tree structures from a given device
btrfs inspect-internal dump-super [options] device [device...]
Dump superblock from a device in a textual form
btrfs inspect-internal tree-stats [options] <device>
Print various stats for trees
btrfs property get [-t <type>] <object> [<name>]
Gets a property from a btrfs object.
btrfs property set [-t <type>] <object> <name> <value>
Sets a property on a btrfs object.
btrfs property list [-t <type>] <object>
Lists available properties with their descriptions for the given object.
btrfs send [-ve] [-p <parent>] [-c <clone-src>] [-f <outfile>] <subvol> [<subvol>...]
Send the subvolume(s) to stdout.
btrfs receive [-ve] [-f <infile>] [--max-errors <N>] <mount>
Receive subvolumes from stdin.
btrfs quota enable <path>
Enable subvolume quota support for a filesystem.
btrfs quota disable <path>
Disable subvolume quota support for a filesystem.
btrfs quota rescan [-sw] <path>
Trash all qgroup numbers and scan the metadata again with the current config.
btrfs qgroup assign [options] <src> <dst> <path>
Assign SRC as the child qgroup of DST
btrfs qgroup remove <src> <dst> <path>
Remove a child qgroup SRC from DST.
btrfs qgroup create <qgroupid> <path>
Create a subvolume quota group.
btrfs qgroup destroy <qgroupid> <path>
Destroy a quota group.
btrfs qgroup show -pcreFf [--sort=qgroupid,rfer,excl,max_rfer,max_excl] <path>
Show subvolume quota groups.
btrfs qgroup limit [options] <size>|none [<qgroupid>] <path>
Set the limits a subvolume quota group.
btrfs replace start [-Bfr] <srcdev>|<devid> <targetdev> <mount_point>
Replace device of a btrfs filesystem.
btrfs replace status [-1] <mount_point>
Print status and progress information of a running device replace
btrfs replace cancel <mount_point>
Cancel a running device replace operation.
btrfs help [--full]
Display help information
btrfs version
Display btrfs-progs version
Use --help as an argument for information on a specific group or command.
usage: mkfs.btrfs [options] dev [ dev ... ]
options:
-A|--alloc-start START the offset to start the FS
-b|--byte-count SIZE total number of bytes in the FS
-d|--data PROFILE data profile, raid0, raid1, raid5, raid6, raid10, dup or single
-f|--force force overwrite of existing filesystem
-l|--leafsize SIZE deprecated, alias for nodesize
-L|--label LABEL set a label
-m|--metadata PROFILE metadata profile, values like data profile
-M|--mixed mix metadata and data together
-n|--nodesize SIZE size of btree nodes
-s|--sectorsize SIZE min block allocation (may not mountable by current kernel)
-r|--rootdir DIR the source directory
-K|--nodiscard do not perform whole device TRIM
-O|--features LIST comma separated list of filesystem features, use '-O list-all' to list features
-U|--uuid UUID specify the filesystem UUID
-q|--quiet no messages except errors
-V|--version print the mkfs.btrfs version and exit
Create a readonly snapshot of '/tmp/snazzer-tests/btrfs-snapshots.working.mnt' in '/tmp/snazzer-tests/btrfs-snapshots.working.mnt/.snapshotz/2016-10-17T130249+0200'
Create a readonly snapshot of '/tmp/snazzer-tests/btrfs-snapshots.working.mnt/srv' in '/tmp/snazzer-tests/btrfs-snapshots.working.mnt/srv/.snapshotz/2016-10-17T130249+0200'
Create a readonly snapshot of '/tmp/snazzer-tests/btrfs-snapshots.working.mnt/srv/s p a c e' in '/tmp/snazzer-tests/btrfs-snapshots.working.mnt/srv/s p a c e/.snapshotz/2016-10-17T130249+0200'
Create a readonly snapshot of '/tmp/snazzer-tests/btrfs-snapshots.working.mnt/home' in '/tmp/snazzer-tests/btrfs-snapshots.working.mnt/home/.snapshotz/2016-10-17T130249+0200'
Create a readonly snapshot of '/tmp/snazzer-tests/btrfs-snapshots.working.mnt/echo `ls "/"; ls /;`; ~!@#$(ls)%^&*()_+-='[]'{}|:<>,./?' in '/tmp/snazzer-tests/btrfs-snapshots.working.mnt/echo `ls "/"; ls /;`; ~!@#$(ls)%^&*()_+-='[]'{}|:<>,./?/.snapshotz/2016-10-17T130249+0200'
Create a readonly snapshot of '/tmp/snazzer-tests/btrfs-snapshots.working.mnt/tmp_thing' in '/tmp/snazzer-tests/btrfs-snapshots.working.mnt/tmp_thing/.snapshotz/2016-10-17T130249+0200'
2 subvolumes excluded in /tmp/snazzer-tests/btrfs-snapshots.working.mnt by /home/florian/Programming/systemautomation/snazzer/tests/data/exclude.patterns.
cat <<EXCL | sudo tee '/tmp/snazzer-tests/btrfs-snapshots.working.mnt/.snapshot_measurements.exclude' >/dev/null
srv
home
var/cache
var/lib/docker/btrfs
echo \`ls "/"; ls /;\`; ~!@#\$(ls)%^&*()_+-='[]'{}|:<>,./?
tmp_thing
EXCL
sudo btrfs subvolume snapshot -r '/tmp/snazzer-tests/btrfs-snapshots.working.mnt' '/tmp/snazzer-tests/btrfs-snapshots.working.mnt/.snapshotz/2016-10-17T130252+0200'
sudo rm '/tmp/snazzer-tests/btrfs-snapshots.working.mnt/.snapshot_measurements.exclude'
cat <<EXCL | sudo tee '/tmp/snazzer-tests/btrfs-snapshots.working.mnt/srv/.snapshot_measurements.exclude' >/dev/null
srv/s p a c e
EXCL
sudo btrfs subvolume snapshot -r '/tmp/snazzer-tests/btrfs-snapshots.working.mnt/srv' '/tmp/snazzer-tests/btrfs-snapshots.working.mnt/srv/.snapshotz/2016-10-17T130252+0200'
sudo rm '/tmp/snazzer-tests/btrfs-snapshots.working.mnt/srv/.snapshot_measurements.exclude'
cat <<EXCL | sudo tee '/tmp/snazzer-tests/btrfs-snapshots.working.mnt/srv/s p a c e/.snapshot_measurements.exclude' >/dev/null
EXCL
sudo btrfs subvolume snapshot -r '/tmp/snazzer-tests/btrfs-snapshots.working.mnt/srv/s p a c e' '/tmp/snazzer-tests/btrfs-snapshots.working.mnt/srv/s p a c e/.snapshotz/2016-10-17T130252+0200'
sudo rm '/tmp/snazzer-tests/btrfs-snapshots.working.mnt/srv/s p a c e/.snapshot_measurements.exclude'
cat <<EXCL | sudo tee '/tmp/snazzer-tests/btrfs-snapshots.working.mnt/home/.snapshot_measurements.exclude' >/dev/null
EXCL
sudo btrfs subvolume snapshot -r '/tmp/snazzer-tests/btrfs-snapshots.working.mnt/home' '/tmp/snazzer-tests/btrfs-snapshots.working.mnt/home/.snapshotz/2016-10-17T130252+0200'
sudo rm '/tmp/snazzer-tests/btrfs-snapshots.working.mnt/home/.snapshot_measurements.exclude'
cat <<EXCL | sudo tee '/tmp/snazzer-tests/btrfs-snapshots.working.mnt/echo `ls "/"; ls /;`; ~!@#$(ls)%^&*()_+-='\''[]'\''{}|:<>,./?/.snapshot_measurements.exclude' >/dev/null
EXCL
sudo btrfs subvolume snapshot -r '/tmp/snazzer-tests/btrfs-snapshots.working.mnt/echo `ls "/"; ls /;`; ~!@#$(ls)%^&*()_+-='\''[]'\''{}|:<>,./?' '/tmp/snazzer-tests/btrfs-snapshots.working.mnt/echo `ls "/"; ls /;`; ~!@#$(ls)%^&*()_+-='\''[]'\''{}|:<>,./?/.snapshotz/2016-10-17T130252+0200'
sudo rm '/tmp/snazzer-tests/btrfs-snapshots.working.mnt/echo `ls "/"; ls /;`; ~!@#$(ls)%^&*()_+-='\''[]'\''{}|:<>,./?/.snapshot_measurements.exclude'
cat <<EXCL | sudo tee '/tmp/snazzer-tests/btrfs-snapshots.working.mnt/tmp_thing/.snapshot_measurements.exclude' >/dev/null
EXCL
sudo btrfs subvolume snapshot -r '/tmp/snazzer-tests/btrfs-snapshots.working.mnt/tmp_thing' '/tmp/snazzer-tests/btrfs-snapshots.working.mnt/tmp_thing/.snapshotz/2016-10-17T130252+0200'
sudo rm '/tmp/snazzer-tests/btrfs-snapshots.working.mnt/tmp_thing/.snapshot_measurements.exclude'
#2 subvolumes excluded in /tmp/snazzer-tests/btrfs-snapshots.working.mnt by /home/florian/Programming/systemautomation/snazzer/tests/data/exclude.patterns.
No idea what's happening there, though, and what they could have done in comprasion to sstephenson/bats to cause this. :disappointed:
Does your mkfs.btrfs not support mkfs.btrfs --help ? Perhaps mkfs.btrfs --version would be a better check for command presence. Or which mkfs.btrfs...
Forgot to mention, /tmp/log does not get created.
Just analysed, my version of mkfs.btrfs --help / btrfs --help return with exit code 0 and print their usage on stdout, the usage information is like it's printed above in the output. Do yours behave differently? This is btrfs-progs 4.7.3, btw.
Regarding the existence test, I'm not sure what's the most clean / compatible / robust way, other alternatives that came to my mind would be bash's integrated hash mkfs.btrfs or also integrated but POSIX-compliant command -v mkfs.btrfs >/dev/null…
(„There should be one-- and preferably only one --obvious way to do it.“ :laughing: )
For testing, I replaced mkfs.btrfs --help with hash mkfs.btrfs. This only cleans up the output as the usage information vanishes, but does not change the result of no /tmp/log and endless waiting after the “#2 subvolumes excluded in …” line.
Yup, I'm also running Arch Linux and get the same error (although on btrfs-progs 4.8.1)
Additionally I have a failing test for snazzer-send-wrapper, don't know if this is related.
✗ snazzer-send-wrapper
(in test file tests/snazzer-send-wrapper.bats, line 18)
`[ "$status" -eq "1" ]' failed
With the work from #43, I managed to get some output!
✗ sudo -n snazzer --list-snapshots '--all' 'foo=" some stuff "' 'hel'\\'' squot '\\''lo' 'asd " dquot " fgh' 'ap ple' ' bon'\\''squot'\\''jour' 'there'
(in test file tests/snazzer-send-wrapper.bats, line 65)
`diff -u $(expected_file) $(actual_file)' failed
--- /tmp/snazzer-tests/snazzer-send-wrapper.bats_7.expected 2016-11-06 00:40:51.458084026 +0100
+++ /tmp/snazzer-tests/snazzer-send-wrapper.bats_7.actual 2016-11-06 00:40:51.478084354 +0100
@@ -1 +1 @@
-10
+/tmp/snazzer-tests/bin/snazzer-send-wrapper (for ) REJECTED: sudo -n snazzer --list-snapshots '--all' 'foo=" some stuff "' 'hel'\\'' squot '\\''lo' 'asd " dquot " fgh' 'ap ple' ' bon'\\''squot'\\''jour' 'there' ### REASON: This should never happen, stopped: '--all' 'foo=" some stuff "' 'hel'\\'' squot '\\''lo' 'asd " dquot " fgh' 'ap ple' ' bon'\\''squot
Okay, according to https://github.com/csirac2/snazzer/pull/43#issuecomment-258670025 @florianjacob's system uses bash for /bin/sh. So it would appear that I've accidentally introduced a dash-ism (the default /bin/sh on debian), or I've done something else that isn't properly emulated by bash in /bin/sh personality.
@Semoar should have bash as /bin/sh, too, as he's on Arch Linux as well.
A dashism seems quite legit, this would explain why it's running fine on travis as well. :laughing:
Ugh, I'm seeing bash interpolating escape \ slightly differently to dash :( What on earth was I thinking when I wrote snazzer-send-wrapper...
EDIT: SSH_ORIGINAL_COMMAND is getting extra slashes... or rather, dash was eating them... from the bats tests?! This is making me insane :)
This is now a stack overflow question :) https://unix.stackexchange.com/questions/321422/why-does-dash-expand-differently-to-bash
Alright, the fix will be to replace much of my current usage of echo with printf, which is more portable in its expansion of escape characters... I'm afraid I've burnt all my snazzer time this weekend on this stupid issue, I will progress the two merge requests next weekend, thanks for your patience.
Created https://github.com/koalaman/shellcheck/issues/771 for shellcheck.net once we have a make test target for that
What on earth was I thinking when I wrote snazzer-send-wrapper...
Probably:
Why do all those mature backup solutions do not leverage btrfs features, particularly copy-on-write snapshots or send/receive transport?!?!??
so exactly what we all were thinking. :wink:
Jokes aside, wasting time on issues that turn out to be simple happens to all of us. :laughing: Until next weekend! Thanks for all your time!