hyperfine

hyperfine copied to clipboard

I would like to compare performance of http://kmkeen.com/jshon/ and https://github.com/jsqry/jsqry-cli2

Both are really fast so hyperfine suggests running with --shell=none for accurate results.

But the latter tool can only process data from stdin therefore I have to call it via shell to provide fixed stdin input like 'jsqry < sample1'. And therefore I cannot reliably measure its performance.

I suggest new parameter to hyperfine --stdin-data=FILEPATH which should open given file as stdin before forking to measured process. This parameter would obviate the need for shell in my test.

I have exactly the same request. I guess the issue is the heterogenity of input passing - some commands read only stdin, some have a --file flag, etc. How to specify which command you wish to redirect the stdin?

Having a --stdin-data (or maybe --input to completement --output) argument in hyperfine sounds reasonable. However, I would first like to understand this use case a bit better.

Both are really fast so hyperfine suggests running with --shell=none for accurate results.

hyperfine suggests this if the commands in question are faster than 5 ms. The reason for this is that a typical shell spawning time is on the order of a few milliseconds. We subtract this shell spawning time from the actual time, but the error being introduced by this procedure is likely to be on a similar order of magnitude.

It is true that using --shell=none will produce more accurate results, as we get rid of this shell spawning (and compensation) procedure. However, please note that most programs themselves also have a startup time of of a few milliseconds. If you (presumably) want to benchmark JSON parsing speed and you see program runtimes smaller than 5 ms, I'm afraid your input is too small.

In this case I would like to compare startup times of these apps, so I have made input is made as small as possible deliberately. Venerable jq got too bloated to start, around 30 ms. So I wanted to compare it with competitors.

I want to compare 'ug ' and 'rg keyword'. but ugrep will exit with non-0. I guess ugrep will check if it's stdin. so --stdin-data is needed.