Performance is constrained by single-core CPU speed when I/O does not depend on CPU

[inga-lovinde]

I couldn't find any issue describing this scenario, so I'll just leave it here, so that at least it's known.

I'm trying to run dua against a remote share (Hetzner storage box mounted over SMB on non-hetzner Linux-based system, to be precise). The file structure is: several millions files, ~10 layers deep, with directories' sizes ranging from 1 to hundreds of subdirectories, and files always being in a leaf directory with 1-2 files (which means there are several millions directories as well).

The problem is that I/O for this remote share has a really high latency, so with ncdu scanning progresses at around 5-10 files per second, and with dust, at maybe 15-30 files per second.

With dua, it seems to scale linearly with the number of threads... at first. So even at 100 threads I'm initially getting better results than at 50 threads, at least in the first seconds of the scan. However, very quickly I get constrained by CPU for some reason, with dua consuming a full single CPU core (more specifically, dua's CPU usage in top seems to hover around 110%, on a modern AMD CPU with eight physical cores). The performance stabilizes at ~1-1.5 million files per hour (or 300-500 files per second). Which is incredibly better than ncdu or dust, but it still feels like dua is not supposed to be single-core-bound in this scenario, not at 500 files (or subdirectories) at least? I understand that non-concurrent parts of the code can be a bottleneck in some scenarios, but maxing out a fast CPU at just 500 files / subdirectories per second does not seem right, so maybe there is something else going on.

(That's on dua v2.26.0 from Alpine package repository.)