hyperfine

hyperfine copied to clipboard

I looked over the documentation, but, unless I'm missing something, I don't see an option for running commands sequentially versus concurrently (ie for load testing). If there is no option for specifying this, I'm assuming that commands run sequentially, is this correct?

Thanks.

That is correct. All benchmark runs are performed sequentially.

Can we reopen this to add support for concurrent executions?

Yes, I can reopen this. Could you please describe your use case in more detail?

I'm really interested about this too.

To describe my need, I'm using hyperfine for performance testing on some endpoint of application, during development. My underlying stack is fully reactive, but we have to take care of blocking thread. So if I run only 1 call at a time, I won't reach the limit of thread allowed (even if I do bad thing in my code). The result in hyperfine will be great and I will continu my development like if everything is ok.

But, in reactive development, the performance drops when we reach a number of concurrent/parallels request at the same time. Because if we don't take care of our thread limitation (or call things which block threads) we will exhaust the thread pool … and the performance will drastically reduce.

For this use case, I think it could be good to provide a way to make concurrent calls from hyperfine.

My use case is simpler, but I think allowing parallel execution could be beneficial.

I am comparing the performance of various versions of the same command line application (there are 4 "switches" that can be independently toggled on/off, thus leading to 16 different cases).

The application is single threaded and performs no network and disk access. I am running each test a number of times (let's say 5) to check for consistency. This results in a total of 80 runs, each several minutes long.

My machine is "beefy" (this statement makes no sense in absolute terms, but is reasonable in this specific case): it has 32 cores and 64 GB of RAM. If I ran with 32 simultaneous processes, my results would not be reliable, because I'd probably end up being thermally limited. However, I am confident that running 2-4 tests in parallel would not hamper the quality of my results too much, while giving a good boost in terms of wall clock time.

An attention point is that only advanced users would know how to safely enable concurrency without making their measurements unreliable: this is clearly a double edged sword.

I do not know if the development effort for implementing this is too much compared to the expected benefit, but I wanted to share a possible use case anyway.

Thanks for this wonderful piece of software!

i too want to benchmark parallel runs;

as a shitty workaround, it's possible to write

hyperfine 'yes "" | head -n5 | parallel -j 5 ./prog'

which should run ./prog 5 times in parallel, and hyperfine measures how long it took to run all 5 invocations in parallel...

(PS: yes it seems -j5 is needed, otherwise parallel may start a different number of processes concurrently.. when i did 100, it only started like 8 without the -j100)