sharkdp
feat: add --aggregate flag to allow "du -sh *"-usage
Hey,
I found this rather old discussion here and wanted to provide here a solution. Since most people (including me) seem to rather look for a replacement of du -sh * instead of implicitly aggregating those, I added a functionality, which makes this behavior configurable. The default behavior now considers all provided path-arguments independently.
I did not dive into optimizing the workload across different workers, so paths are just worked on sequentially, but while doing so, parallelized.
Thank you for working on this.
I did not dive into optimizing the workload across different workers, so paths are just worked on sequentially, but while doing so, parallelized.
I think we should at least perform some benchmarks. It's easy to imagine that this will result in a performance regression for some workloads.
Alright, so I found the time to benchmark this. Thanks for your instructions in the README. Made my life a lot easier!
diskus-Benchmark for PR57
x86 WSL - Debian 11
Cold disk cache
Benchmark 1: diskus
Time (mean ± σ): 308.5 ms ± 18.6 ms [User: 40.0 ms, System: 256.5 ms]
Range (min … max): 284.2 ms … 350.6 ms 10 runs
Benchmark 2: du -sh
Time (mean ± σ): 831.8 ms ± 35.6 ms [User: 2.1 ms, System: 70.5 ms]
Range (min … max): 792.0 ms … 885.8 ms 10 runs
Benchmark 3: sn p -d0 -j8
Time (mean ± σ): 361.0 ms ± 7.3 ms [User: 385.3 ms, System: 86.1 ms]
Range (min … max): 349.4 ms … 373.6 ms 10 runs
Benchmark 4: dust -d0
Time (mean ± σ): 1.302 s ± 0.014 s [User: 0.062 s, System: 0.993 s]
Range (min … max): 1.288 s … 1.323 s 10 runs
Summary
diskus ran
1.17 ± 0.07 times faster than sn p -d0 -j8
2.70 ± 0.20 times faster than du -sh
4.22 ± 0.26 times faster than dust -d0
Warm disk cache
Benchmark 1: diskus
Time (mean ± σ): 261.6 ms ± 14.0 ms [User: 21.0 ms, System: 240.0 ms]
Range (min … max): 231.1 ms … 283.2 ms 12 runs
Benchmark 2: du -sh
Time (mean ± σ): 773.3 ms ± 14.0 ms [User: 3.1 ms, System: 59.8 ms]
Range (min … max): 748.5 ms … 792.9 ms 10 runs
Benchmark 3: sn p -d0 -j8
Time (mean ± σ): 343.5 ms ± 4.3 ms [User: 390.0 ms, System: 92.0 ms]
Range (min … max): 337.0 ms … 350.1 ms 10 runs
Benchmark 4: dust -d0
Time (mean ± σ): 341.9 ms ± 6.9 ms [User: 42.2 ms, System: 192.3 ms]
Range (min … max): 334.1 ms … 358.5 ms 10 runs
Summary
diskus ran
1.31 ± 0.07 times faster than dust -d0
1.31 ± 0.07 times faster than sn p -d0 -j8
2.96 ± 0.17 times faster than du -sh
MacOS-15 - ARM64
Cold disk cache
not possible, due to missing admin permissions on the device
Warm disk cache
Benchmark 1: diskus
Time (mean ± σ): 7.0 ms ± 1.5 ms [User: 5.2 ms, System: 48.7 ms]
Range (min … max): 6.1 ms … 21.0 ms 288 runs
Benchmark 2: du -sh
Time (mean ± σ): 10.9 ms ± 0.3 ms [User: 0.9 ms, System: 9.9 ms]
Range (min … max): 10.1 ms … 12.4 ms 206 runs
Benchmark 3: sn p -d0 -j8
Time (mean ± σ): 9.9 ms ± 0.4 ms [User: 2.7 ms, System: 13.4 ms]
Range (min … max): 9.1 ms … 12.5 ms 218 runs
Benchmark 4: dust -d0
Time (mean ± σ): 8.3 ms ± 0.6 ms [User: 6.2 ms, System: 39.5 ms]
Range (min … max): 7.3 ms … 10.6 ms 259 runs
Summary
diskus ran
1.18 ± 0.27 times faster than dust -d0
1.42 ± 0.31 times faster than sn p -d0 -j8
1.56 ± 0.35 times faster than du -sh
x86 Ubuntu 24
Cold disk cache
Benchmark 1: diskus
Time (mean ± σ): 327.1 ms ± 3.5 ms [User: 444.0 ms, System: 1155.8 ms]
Range (min … max): 320.5 ms … 332.3 ms 10 runs
Benchmark 2: du -sh
Time (mean ± σ): 2.508 s ± 0.096 s [User: 0.187 s, System: 0.816 s]
Range (min … max): 2.306 s … 2.606 s 10 runs
Benchmark 3: sn p -d0 -j8
Time (mean ± σ): 1.144 s ± 0.049 s [User: 0.135 s, System: 0.777 s]
Range (min … max): 1.085 s … 1.241 s 10 runs
Benchmark 4: dust -d0
Time (mean ± σ): 1.706 s ± 0.123 s [User: 0.368 s, System: 2.384 s]
Range (min … max): 1.647 s … 2.052 s 10 runs
Summary
diskus ran
3.50 ± 0.15 times faster than sn p -d0 -j8
5.21 ± 0.38 times faster than dust -d0
7.67 ± 0.30 times faster than du -sh
x86 Ubuntu 24
Warm disk cache
Benchmark 1: diskus
Time (mean ± σ): 90.2 ms ± 3.4 ms [User: 237.7 ms, System: 454.6 ms]
Range (min … max): 86.2 ms … 100.9 ms 32 runs
Benchmark 2: du -sh
Time (mean ± σ): 364.4 ms ± 1.2 ms [User: 128.8 ms, System: 235.8 ms]
Range (min … max): 362.8 ms … 366.0 ms 10 runs
Benchmark 3: sn p -d0 -j8
Time (mean ± σ): 133.0 ms ± 4.4 ms [User: 93.6 ms, System: 228.9 ms]
Range (min … max): 125.9 ms … 138.7 ms 23 runs
Benchmark 4: dust -d0
Time (mean ± σ): 240.6 ms ± 11.7 ms [User: 335.3 ms, System: 795.3 ms]
Range (min … max): 229.7 ms … 267.7 ms 12 runs
Summary
diskus ran
1.47 ± 0.07 times faster than sn p -d0 -j8
2.67 ± 0.16 times faster than dust -d0
4.04 ± 0.15 times faster than du -sh