Why Does the FIO Performance Degrade in the Last 15s+(from 59w IOPS to 5w)When the FIO Is Executed for 1 Min?

[ssandl]

Please acknowledge you have done the following before creating a ticket

• [x] I have read the GitHub issues section of REPORTING-BUGS.

fio commond: fio -directory=/mnt/fio/ -direct=1 -iodepth 1 -thread -rw=randwrite -ioengine=psync -bs=8k -size=5G -numjobs=100 -runtime=60 -group_reporting -name=rand_100write_8k

At start: IOPS: 61w cpu%: 958 MEM:0.1%

last 15s： IOPS：5w CPU%：0 MEM：0.1%

I don't know why the fio CPU usage cannot increase in the last few seconds. In addition, the disk latency does not increase by 10 times, maybe increase by about 2 times.

[ssandl]

The disk is an NVMe disk and is not full.

[sitsofe]

Hello @ssandl: Difficult to say without more information e.g. is the sustained workload forcing your SSD getting behind on its garbage collection?. I think we would need to know at least the following:

• Does it happen as soon when you've trimmed/discarded the SSD?
• Does it happen with fewer (e.g. 8 or less but the fewer the better) jobs?
• If you extend the runtime does it happen after the same amount of time or is it always 15 seconds from the end?
• Similarly if you use write_*_log options (see https://fio.readthedocs.io/en/latest/fio_doc.html#cmdoption-arg-write_bw_log and below) do you get insight into just when the IOPS go down and the latency goes up?
• It would be useful to see the fio's summary output

[sitsofe]

Closing due to lack of reply from reporter. If this issue is still happening with the latest fio (see https://github.com/axboe/fio/releases to find out which version that is) please reopen. Thanks!