wg
Added output as JSON
Hello I added JSON output option with -j / --json-output.
Other outputs are disabled if JSON is enabled ande vice-versa.
@all Please don't hurry. @ZuluPro sample output helps to review output format.
This is duplicate of #72 and there are no describe about why report.lua is not enough.
Addtionally, I expect more detailed information for JSON. I don't expect "for human" information (like avg) in JSON. For example, someone may want histogram instead of percentile.
If wrk has -J option, people would expect backward compatibility. It's difficult to refactoring JSON format.
So I think report.lua is much better. It is customizable. It's easier to keep backward compatibility, because people can keep using old report even when wrk updated it's report.lua format.
Hello @methane
@ZuluPro sample output helps to review output format.
Here's an example of output:
{
"url": "https://example.com",
"threads": 1,
"connections": 1,
"duration": 2,
"timeout": 1000,
"script": null,
"runtime_us": 2008747,
"bytes": 241470,
"bytes_per_sec": 120209.264780,
"requests_count": 3,
"requests_per_sec": 1.493468,
"requests_mean": 1.666667,
"requests_stdev": 1.527525,
"requests_min": 0,
"requests_max": 3,
"requests_within_stdev": 66.67,
"latency_mean": 0.399244,
"latency_stdev": 0.095570,
"latency_min": 0.329713,
"latency_max": 0.508223,
"latency_within_stdev": 66.67,
"latency_percentile_50": 0.36,
"latency_percentile_75": 0.51,
"latency_percentile_90": 0.51,
"latency_percentile_95": 0.51,
"latency_percentile_99": 0.51,
"errors_connect": 0,
"errors_read": 0,
"errors_write": 0,
"errors_timeout": 0,
"errors_status": 0
}
This is duplicate of #72 ...
Your link is pointing to an issue, not a pull request, but yes, there were several (closed) contributions about JSON output:
- #17 : Basically the same approach than me The closing comment says "should be easy enough to process with something like an awk script." It is not as the metrics can changes (nsec, msec, sec or k, m, g)
- #305 : JSON with LuA Apparently, it should not be merged as a community is coming
- #361 : JSON with LuA Still opened without response
... and there are no describe about why report.lua is not enough.
report.lua gives only a poor latency distribution, if you look at my JSON output there are tons more:
- Run configuration
- Requests data
- Latency data
- Errors
Addtionally, I expect more detailed information for JSON. I don't expect "for human" information (like avg) in JSON. For example, someone may want histogram instead of percentile.
This is a first contribution of JSON output and it actually gives more than the existing text one with more accuracy. Your example of histogram is a totally other feature/PR. Otherwise it could be enabled with the --latency option.
If wrk has -J option, people would expect backward compatibility. It's difficult to refactoring JSON format.
It's not hard if you warn users. People could expect changement if they change major or minor version.
So I think report.lua is much better. It is customizable. It's easier to keep backward compatibility, because people can keep using old report even when wrk updated it's report.lua format.
report.lua is not comparable, as said, it's a basis to make custom report with LuA. But except programmers, people expect a command working out-of-the-box. If I am a sysadmin I don't expect to learn LuA just to have machine-readable data.
because people can keep using old report even when wrk updated it's report.lua format.
Solutions aren't mutually exclusive, the ability to use LuA isn't erased by the JSON option
report.luagives only a poor latency distribution, if you look at my JSON output there are tons more:
"report.lua" is just an example. You can propose adding "reportjson.lua" example. Then people can customize it for their usage.
Your link is pointing to an issue, not a pull request, but yes, there were several (closed) contributions about JSON output:
...
report.luais not comparable, as said, it's a basis to make custom report with LuA. But except programmers, people expect a command working out-of-the-box. If I am a sysadmin I don't expect to learn LuA just to have machine-readable data.
...
Solutions aren't mutually exclusive, the ability to use LuA isn't erased by the JSON option
I meant you should describe these reasoning before sending PR with only "Hello I added JSON output option", and pinging. Especially for features already rejected in the past. I don't oppose against this. (+0)
@methane thank you for taking the time to review and comment on this PR!
@ZuluPro as methane points out it should be possible to generate a JSON report using the Lua scripting API. This is preferable to hard-coding reports in wrk itself because everyone will have different requirements for format, content, etc.
My recommendation would be to develop JSON reporting as a Lua script and publish it as a blog post or to your own GitHub repo. I've left this repo's wiki open and it'd be a great contribution if you wanted to curate a page about available scripts and including yours.
Hello @wg
Except the latency details, the output gives almost everything, so no one could ask more data. If someone has issue with format it would be JSON's keys or unit of measurement: nothing hard as JSON is typed and machine readable.
I would like to point the project's audience : Benchmarkers. You are asking for LuA programming just to be able to have machine-readable output. For a sysadmin, your project stay a handy tool, but just for human eyes.
@methane
"report.lua" is just an example. You can propose adding "reportjson.lua" example. Then people can customize it for their usage.
As said, people who aren't programmers won't use scripts. Seeing reaction on top of this PR, I think a lot of people are agreed.
I meant you should describe these reasoning before sending PR with only "Hello I added JSON output option", and pinging. Especially for features already rejected in the past.
I saw the other PR after did mine, seeing the popularity of this project I didn't think this feature could be already rejected. About reason, nowadays, it's obvious that a machine-readable output is a base requirements for a benchmark command.
I tried to implement sample implementation of JSON output.
https://gist.github.com/methane/8bb4fedf27e4d13db80078302f8954da
As you can see, it's difficult to combine with tools like jq by pipe. Both of text output and Lua output are written into stdout.
@methane
As you can see, it's difficult to combine with tools like jq by pipe.
Yes this is why my -j option disables the default output. Text is useless as we aim machines.
Does the lua environment allow to disable the default output ?
I'd suggest writing JSON output to a file whose path is specified via a script arg. Most likely someone who wants machine-readable output is also invoking wrk with a script. Easy enough to record the command line parameters in that script, generate a temp file name, pass that to wrk and read the results if wrk completes successfully.
I'd suggest writing JSON output to a file whose path is specified via a script arg.
@wg But with your suggestion we lose the ability to just pipe to another command, i.e. a JSON parser.
As what I coded is an option, it doesn't change the default behavior, but the new output should be also be given in usual conditions.
Yes this is why my
-joption disables the default output. Text is useless as we aim machines.
If so, why didn't describe it from first? You should describe "why this is needed" when (or before) sending pull request.
@wg But with your suggestion we lose the ability to just pipe to another command, i.e. a JSON parser.
Using pipe is not a strict requirement for me.
@methane good question, it's a little tricky since done runs in a different context than init, need to grab a reference to one thread in setup, store the arg in a global in init, then use thread:get.
I'd use some Lua JSON library like https://github.com/rxi/json.lua to avoid a lot of ugly manual JSON building and then a script like:
json = require "json"
function setup(thread)
thread0 = thread0 or thread
end
function init(args)
file = args[1] or "/dev/null"
end
function done(summary, latency, requests)
file = io.open(thread0:get("file"), "w")
percentiles = {}
for _, p in pairs({ 50, 90, 99, 99.999 }) do
k = string.format("%g%%", p)
v = latency:percentile(p)
percentiles[k] = v
end
file:write(json.encode({
duration = summary.duration,
requests = summary.requests,
bytes = summary.bytes,
errors = summary.errors,
latency = {
min = latency.min,
max = latency.max,
mean = latency.mean,
stdev = latency.stdev,
percentiles = percentiles,
},
}))
file:close()
end
Run as wrk ... -s report.lua -- output.json
Wow, good example! Would you post it in Wiki? I think it is very helpful for many users.
I also need this option. It would really helped to make atomated benchmark I am working on now.
Lua script is highly customisable, but it not helps. It just makes everything more complicated (
Why messing with additional files, when you could add just a small parameter to command? It would be so much easier...
