Abysmal performance with default per_txg_dirty_frees_percent value when doing -j 104 buildkernel on FreeBSD

[mjguzik]

I'm running FreeBSD with zfs as of:

commit a582d52993134905f2e96cebe7aff410f29b1ad8 Author: Umer Saleem [email protected] Date: Thu Aug 25 02:20:43 2022 +0500

Updates for snapshots_changed property

I found -j 104 buildkernel performance to be incredibly bad, about twice the time needed by tmpfs and about 1.5 the time needed by ufs. I tracked it down to enormous off cpu time spent waiting on transactions to finish, almost all of which comes from waiting for nodes to be removed.

Check the flamegraph: https://people.freebsd.org/~mjg/zfs-buildkernel-offcpu.svg

I got pointed by @allanjude at vfs.zfs.per_txg_dirty_frees_percent and it indeed is the culprit.

The parameter currently sits at 5, changed from 30 here:

commit 65282ee9e06b130f1f0169baf5d9bf0dd8fc1ef9 Author: Alek P [email protected] Date: Tue Feb 12 12:01:08 2019 -0800

Freeing throttle should account for holes

It may be its significance has changed(?).

Anyhow, this is what I get when running -j 104 buildkernel (total real time varies by few seconds for the longer cases between runs, I don't think it matters for this test):

tunable	time
0	2564.69s user 440.50s system 8546% cpu 35.163 total
1	2314.02s user 935.17s system 2738% cpu 1:58.65 total
5	2538.62s user 528.88s system 4478% cpu 1:08.49 total
10	2568.74s user 479.62s system 6869% cpu 44.374 total
15	2567.89s user 488.04s system 7946% cpu 38.458 total
20	2554.29s user 539.25s system 8321% cpu 37.176 total
25	2558.83s user 495.73s system 8595% cpu 35.536 total

As you can see 25 gives roughly the same performance as 0 (no throttling).

I don't know what values makes the most sense by default, the current one is most definitely bad.

The drive is Micron M600 MTFDDAK1T0MBF MU03, pool has default params + noatime.

May I suggest other people bench their favorite workload(tm) with different parameters?