Regression: "invalid flags combined with -t" when resuming with `syncoid --sendoptions`

[Deltik]

Bug Description

Syncoid with --sendoptions now fails with the error invalid flags combined with -t when resuming an interrupted transfer:

root@demo:~# syncoid --sendoptions="pw" jimsalterjrs_sanoid_918/{source,destination}
INFO: Resuming interrupted zfs send/receive from jimsalterjrs_sanoid_918/source to jimsalterjrs_sanoid_918/destination (~ 8.1 MB remaining):
invalid flags combined with -t
usage:
        send [-DnPpRvLecwhb] [-[i|I] snapshot] <snapshot>
        send [-DnvPLecw] [-i snapshot|bookmark] <filesystem|volume|snapshot>
        send [-DnPpvLec] [-i bookmark|snapshot] --redact <bookmark> <snapshot>
        send [-nvPe] -t <receive_resume_token>
        send [-Pnv] --saved filesystem

For the property list, run: zfs set|get

For the delegated permission list, run: zfs allow|unallow
0.00 B 0:00:00 [0.00 B/s] [>                                                                                                                                                                                                                                               ]  0%            
cannot receive: failed to read from stream
CRITICAL ERROR:  zfs send -p -w  -t 1-1694168810-138-789c636064000310a501c49c50360710a715e5e7a69766a6304081c717d7729790471a0a40363b92bafca4acd4e412060626a83a0cf9b4b4e2d412904c0d5c9e0d493ea9b224b518484f08aec3aabf241fe28a7d1dcb2eddf7f9a9ef8124cf0996cf4bcc4d6560c8cacc2d4ecc29492dca2a2a8e2f4ecccbcf4c89b734b4d02fce2f2d4a4e7528cecf4d2dc9c8cc4b87f947026a3eccffa9b949a929f9d960fbb991c493f3730b8a528b8b81527000002ebc32a1 | mbuffer  -q -s 128k -m 16M | pv -p -t -e -r -b -s 8536288 |  zfs receive  -s -F 'jimsalterjrs_sanoid_918/destination' 2>&1 failed: 256

This is a regression introduced by commit 09b42d6ade2843171be810db7df3b189954b7992:

root@demo:~/sanoid# git bisect log
git bisect start
# bad: [b31ed6e325e4e145d112fab582e348f6ca90631b] Merge pull request #916 from 0xFelix/zstdmt
git bisect bad b31ed6e325e4e145d112fab582e348f6ca90631b
# good: [9564454d74d9b8bc2b924ee261a6e40c5c278662] Merge pull request #601 from phreaker0/prepare-2.1.0
git bisect good 9564454d74d9b8bc2b924ee261a6e40c5c278662
# good: [3bde229a617b2fcb9976e8837d09b6e136df07c2] Merge pull request #622 from rbike/syncoid-hold
git bisect good 3bde229a617b2fcb9976e8837d09b6e136df07c2
# bad: [790ea544ff82bb4945c269b3fe6e451076d6307b] Merge branch 'master' into zfs-get
git bisect bad 790ea544ff82bb4945c269b3fe6e451076d6307b
# good: [18ccb7df350d22bd02a4efd01e4fefd1db4bdc3f] Fix tiny typo in README.md
git bisect good 18ccb7df350d22bd02a4efd01e4fefd1db4bdc3f
# bad: [ca6e60b920994b5a61b295674121aa23f32dc0bb] Merge branch 'master' into update-send-recv-options
git bisect bad ca6e60b920994b5a61b295674121aa23f32dc0bb
# bad: [14ed85163a82c743d67c2b25fac1882f302ab0ea] Filter snapshots in getsnapsfallback()
git bisect bad 14ed85163a82c743d67c2b25fac1882f302ab0ea
# bad: [603c286b50128a3ceb061f0e2c03310a00220977] Don't iterate over snaps twice
git bisect bad 603c286b50128a3ceb061f0e2c03310a00220977
# bad: [09b42d6ade2843171be810db7df3b189954b7992] Refactor system calls
git bisect bad 09b42d6ade2843171be810db7df3b189954b7992
# good: [c4e70280225f7b3ec03cc075daf218dde468412f] Refactor terminal output
git bisect good c4e70280225f7b3ec03cc075daf218dde468412f
# first bad commit: [09b42d6ade2843171be810db7df3b189954b7992] Refactor system calls

How to Reproduce

I hacked together a git bisect run script that finds the regression:

#!/bin/bash

truncate -s128M /tmp/jimsalterjrs_sanoid_918.img
zpool create jimsalterjrs_sanoid_918 /tmp/jimsalterjrs_sanoid_918.img
zfs create jimsalterjrs_sanoid_918/source
zfs snap jimsalterjrs_sanoid_918/source@empty
dd if=/dev/urandom of=/jimsalterjrs_sanoid_918/source/garbage.bin bs=1M count=16
zfs snap jimsalterjrs_sanoid_918/source@something

zfs send -pwR jimsalterjrs_sanoid_918/source@something | head --bytes=8M | zfs recv -s jimsalterjrs_sanoid_918/destination

./syncoid --sendoptions="pw" jimsalterjrs_sanoid_918/source jimsalterjrs_sanoid_918/destination

if [ $? -ne 0 ]; then
    echo "Regression detected"
    zfs destroy -r jimsalterjrs_sanoid_918/destination
    zpool destroy jimsalterjrs_sanoid_918
    exit 1
else
    echo "No regression detected"
    zfs destroy -r jimsalterjrs_sanoid_918/destination
    zpool destroy jimsalterjrs_sanoid_918
    exit 0
fi

Manual run:

root@demo:~# truncate -s128M /tmp/jimsalterjrs_sanoid_918.img
root@demo:~# zpool create jimsalterjrs_sanoid_918 /tmp/jimsalterjrs_sanoid_918.img
root@demo:~# zfs create jimsalterjrs_sanoid_918/source
root@demo:~# zfs snap jimsalterjrs_sanoid_918/source@empty
root@demo:~# dd if=/dev/urandom of=/jimsalterjrs_sanoid_918/source/garbage.bin bs=1M count=16
16+0 records in
16+0 records out
16777216 bytes (17 MB, 16 MiB) copied, 0.303385 s, 55.3 MB/s
root@demo:~# zfs snap jimsalterjrs_sanoid_918/source@something
root@demo:~# zfs send -pwR jimsalterjrs_sanoid_918/source@something | head --bytes=8M | zfs recv -s jimsalterjrs_sanoid_918/destination
cannot receive incremental stream: checksum mismatch or incomplete stream.
Partially received snapshot is saved.
A resuming stream can be generated on the sending system by running:
    zfs send -t 1-1694168810-138-789c636064000310a501c49c50360710a715e5e7a69766a6304081c717d7729790471a0a40363b92bafca4acd4e412060626a83a0cf9b4b4e2d412904c0d5c9e0d493ea9b224b518484f08aec3aabf241fe28a7d1dcb2eddf7f9a9ef8124cf0996cf4bcc4d6560c8cacc2d4ecc29492dca2a2a8e2f4ecccbcf4c89b734b4d02fce2f2d4a4e7528cecf4d2dc9c8cc4b87f947026a3eccffa9b949a929f9d960fbb991c493f3730b8a528b8b81527000002ebc32a1
root@demo:~# syncoid --sendoptions="pw" jimsalterjrs_sanoid_918/{source,destination}
Resuming interrupted zfs send/receive from jimsalterjrs_sanoid_918/source to jimsalterjrs_sanoid_918/destination (~ 8.1 MB remaining):
8.28MiB 0:00:00 [ 118MiB/s] [=============================================================================================================================================================================================================================================] 101%            
Sending incremental jimsalterjrs_sanoid_918/source@something ... syncoid_demo_2024-04-24:17:39:27-GMT00:00 (~ 4 KB):
2.23KiB 0:00:00 [79.3KiB/s] [=================================================================================================================================>                                                                                                            ] 55%            
root@demo:~# zfs destroy -r jimsalterjrs_sanoid_918/destination
root@demo:~# zfs send -pwR jimsalterjrs_sanoid_918/source@something | head --bytes=8M | zfs recv -s jimsalterjrs_sanoid_918/destination
cannot receive incremental stream: checksum mismatch or incomplete stream.
Partially received snapshot is saved.
A resuming stream can be generated on the sending system by running:
    zfs send -t 1-1694168810-138-789c636064000310a501c49c50360710a715e5e7a69766a6304081c717d7729790471a0a40363b92bafca4acd4e412060626a83a0cf9b4b4e2d412904c0d5c9e0d493ea9b224b518484f08aec3aabf241fe28a7d1dcb2eddf7f9a9ef8124cf0996cf4bcc4d6560c8cacc2d4ecc29492dca2a2a8e2f4ecccbcf4c89b734b4d02fce2f2d4a4e7528cecf4d2dc9c8cc4b87f947026a3eccffa9b949a929f9d960fbb991c493f3730b8a528b8b81527000002ebc32a1
root@demo:~# syncoid --sendoptions="pw" jimsalterjrs_sanoid_918/{source,destination}
INFO: Resuming interrupted zfs send/receive from jimsalterjrs_sanoid_918/source to jimsalterjrs_sanoid_918/destination (~ 8.1 MB remaining):
invalid flags combined with -t
usage:
        send [-DnPpRvLecwhb] [-[i|I] snapshot] <snapshot>
        send [-DnvPLecw] [-i snapshot|bookmark] <filesystem|volume|snapshot>
        send [-DnPpvLec] [-i bookmark|snapshot] --redact <bookmark> <snapshot>
        send [-nvPe] -t <receive_resume_token>
        send [-Pnv] --saved filesystem

For the property list, run: zfs set|get

For the delegated permission list, run: zfs allow|unallow
0.00 B 0:00:00 [0.00 B/s] [>                                                                                                                                                                                                                                               ]  0%            
cannot receive: failed to read from stream
CRITICAL ERROR:  zfs send -p -w  -t 1-1694168810-138-789c636064000310a501c49c50360710a715e5e7a69766a6304081c717d7729790471a0a40363b92bafca4acd4e412060626a83a0cf9b4b4e2d412904c0d5c9e0d493ea9b224b518484f08aec3aabf241fe28a7d1dcb2eddf7f9a9ef8124cf0996cf4bcc4d6560c8cacc2d4ecc29492dca2a2a8e2f4ecccbcf4c89b734b4d02fce2f2d4a4e7528cecf4d2dc9c8cc4b87f947026a3eccffa9b949a929f9d960fbb991c493f3730b8a528b8b81527000002ebc32a1 | mbuffer  -q -s 128k -m 16M | pv -p -t -e -r -b -s 8536288 |  zfs receive  -s -F 'jimsalterjrs_sanoid_918/destination' 2>&1 failed: 256
root@demo:~# echo $?
2

Expected Behavior

The unsupported syncoid --sendoptions are removed, allowing interrupted send/receive operations to complete.

In Sanoid v2.1.0, the output looked like this:

root@demo:~# syncoid --sendoptions="pw" jimsalterjrs_sanoid_918/{source,destination}
Resuming interrupted zfs send/receive from jimsalterjrs_sanoid_918/source to jimsalterjrs_sanoid_918/destination (~ 8.1 MB remaining):
8.28MiB 0:00:00 [ 118MiB/s] [=============================================================================================================================================================================================================================================] 101%            
Sending incremental jimsalterjrs_sanoid_918/source@something ... syncoid_demo_2024-04-24:17:39:27-GMT00:00 (~ 4 KB):
2.23KiB 0:00:00 [79.3KiB/s] [=================================================================================================================================>                                                                                                            ] 55%
root@demo:~# echo $?
0