fio-3.41 regression: verify fails with "bad magic header 0, wanted acca" when using io_uring_cmd + verification

[BalkiratS]

Please acknowledge the following before creating a ticket

• [x] I have read the GitHub issues section of REPORTING-BUGS.

Description of the bug: Starting with fio-3.41, the built-in verification feature (verify=crc32, do_verify=1, verify_state_save=1 / verify_state_load=1) no longer works when ioengine=io_uring_cmd is used.

The write phase still completes successfully and creates the verify-state files, but any subsequent verify_only pass fails with errors like:

verify: bad magic header 0, wanted acca ...
fio: pid=xxxx, err=84/file:io_u.c:2318, func=io_u_queued_complete, error=Invalid or incomplete multibyte or wide character

The exact same job files, same drives, same kernel, and same host work perfectly with fio-3.40. This regression is introduced in fio-3.41.

Environment:

• CentOS Stream 9
• Kernel 6.13.2-0
• x86_64
• Any NVMe device with lbaf 4k+64 and accessed via /dev/ngXn1

fio version:

• fio-3.40 → verification works perfectly
• fio-3.41 → verification fails (regression)

Reproduction steps

Job files: write.fio

[global]
rw=write
blocksize=512k
md_per_io_size=8k
iodepth=64
ioengine=io_uring_cmd
cmd_type=nvme
verify=crc32
time_based
runtime=120
do_verify=1
verify_state_save=1
verify_async=4

[job0]
filename=/dev/ng2n1
new_group=1

[job1]
filename=/dev/ng3n1
new_group=1

verify.fio

[global]
rw=write
blocksize=512k
md_per_io_size=8k
iodepth=64
ioengine=io_uring_cmd
cmd_type=nvme
verify=crc32
verify_only
verify_state_load=1

[job0]
filename=/dev/ng2n1
new_group=1

[job1]
filename=/dev/ng3n1
new_group=1

Commands:

# With fio-3.40 → everything works
fio write.fio      # succeeds, creates state files
fio verify.fio     # succeeds, no errors

# With fio-3.41 → regression
fio write.fio      # still succeeds
fio verify.fio     # fails with the errors shown above

Actual output with fio-3.41 (verify phase, few lines only – thousands more follow):

$ fio verify_2.fio
job0: (g=0): rw=write, bs=(R) 512KiB-512KiB, (W) 512KiB-512KiB, (T) 512KiB-512KiB, ioengine=io_uring_cmd, iodepth=64
job1: (g=1): rw=write, bs=(R) 512KiB-512KiB, (W) 512KiB-512KiB, (T) 512KiB-512KiB, ioengine=io_uring_cmd, iodepth=64
fio-3.41
Starting 2 processes
verify: bad magic header 0, wanted acca at file /dev/ng3n1 offset 44027084800, length 524288 (requested block: offset=44027084800, length=524288)
verify: bad magic header 0, wanted acca at file /dev/ng3n1 offset 44027609088, length 524288 (requested block: offset=44027609088, length=524288)
.
.
.
fio: pid=3153349, err=84/file:io_u.c:2318, func=io_u_queued_complete, error=Invalid or incomplete multibyte or wide character

With fio-3.40 the verify pass is completely clean and exits with status 0.

[sitsofe]

Hi @BalkiratS, Are you able to reproduce this with other more generic ioengines and less options (e.g. io_uring or pvsync2)?

[BalkiratS]

@sitsofe Using ioengine=io_uring the verification is still failing with this error:

$fio verify.fio
job0: (g=0): rw=write, bs=(R) 128KiB-128KiB, (W) 128KiB-128KiB, (T) 128KiB-128KiB, ioengine=io_uring, iodepth=64
job1: (g=1): rw=write, bs=(R) 128KiB-128KiB, (W) 128KiB-128KiB, (T) 128KiB-128KiB, ioengine=io_uring, iodepth=64
fio-3.41
Starting 2 processes
verify: unsupported header version 4, wanted 81. Are you trying to verify across versions of fio? at file /dev/nvme3n1 offset 36892049408, length 131072 (requested block: offset=36892049408, length=131072)
verify: unsupported header version 4, wanted 81. Are you trying to verify across versions of fio? at file /dev/nvme3n1 offset 36892180480, length 131072 (requested block: offset=36892180480, length=131072)

[sitsofe]

Hi @BalkiratS: I think this has been fixed by https://github.com/axboe/fio/commit/466a2e664d28756c61e03bdc353c683427ab9870 on master because the following works for me with when I check out 26c77cd375fc48511fda408e924416746ea878af:

./fio --size=128k --filename=/tmp/fio.tmp --rw=write --verify=crc32c --runtime=2s --verify_state_save=1 --rate_iops=1 --name=vgo
[...]
./fio --size=128k --filename=/tmp/fio.tmp --rw=write --verify=crc32c --runtime=2s --verify_state_save=1 --rate_iops=1 --name=vgo --verify_only --verify_state_load=1
[...]
fio-3.41-51-g26c77
Starting 1 process
Stop verify because seq 2 >= 2

vgo: (groupid=0, jobs=1): err= 0: pid=10823: Tue Dec  9 10:24:59 2025

Can you retry with current master and see if your issue still persists?

[veenums-sudo]

[global] verify_dump=1 verify_fatal=1 exitall_on_error=1 rw=write direct=1 ioengine=libaio size=2G group_reporting randrepeat=0 bs=4K iodepth=16 numjobs=50 directory=/mnt/sandbox/pfs verify=crc32c fallocate=truncate
startdelay=5,60 verify_backlog=10 verify_dump=1
[seqwritetest] iodepth_batch=0 stonewall

Seeing seqwritetest: (groupid=0, jobs=50): err=84 (file:io_u.c:2318, func=io_u_queued_complete, error=Invalid or incomplete multibyte or wide character)

fio version = 3.41

[sitsofe]

@veenums-sudo: Your account appears to be new so you may be unaware of the best way to report an issue. You've jumped on issues raised by somebody else twice (here and https://github.com/axboe/fio/issues/1517#issuecomment-3649181761 ) but it's unclear whether what you're seeing is the same as the original reporters (for example the error message you're reporting is different). Further the other issue has been closed for two years. In general when working with projects other than your own, if there's any doubt over whether your issue really is identical to someone else's it is good etiquette to open a brand new issue and include a reference the other issue so that you avoid creating confusion and avoid disturbing someone unnecessarily.

Some more recommendations:

• The original reporter on this issue has shown their issue was introduced sometime between fio 3.40 and fio 3.41. This is valuable information and again if your issue doesn't follow the same behaviour it is likely different. Does your issue follow the same pattern?
• You didn't include all the information that makes diagnosing your issue easier (see https://github.com/axboe/fio/blob/master/REPORTING-BUGS for the bare minimum). Notice how the original reporter filled out a bug template in the first comment thus helping the reader. The reporter even took time to use markdown formatting to make the report easier to read. All this helps the potential reader.
• Don't file the same thing multiple times in different places - stick to just one. Failing to do this may end up wasting the time of multiple different people who are unaware of the other has said.

Now before you go further I'm going to ask you: does your (@veenums-sudo) problem happen on a build of the lastest master version of fio? If it does I recommend filing a new issue and structuring it as well as the report that you see in the first comment here. If it doesn't, post a comment here so anyone else who arrives here knows how you resolved it.