[Proposal] Add envoyproxy/nighthawk's client as a load generator

[oschaaf]

Inspired by https://github.com/envoyproxy/envoy/issues/13412, I took a look at this repo, and at the surface it seems to me that it would take a relatively small effort to introduce a wrapper for using Nighthawk's load gen client [1].

If there's interest in a pull request, I'd be up for contributing this.

Nighthawk's offers some options and knobs that may be nice to have (for example explicit connection re-use strategies, open-loop mode, a gRPC service for remote load test execution).

[1] https://github.com/envoyproxy/nighthawk

[sebastienros]

That's interesting. Obviously Bombardier is more popular, and even though we also have a wrk implementation, Bombardier is the most similar here as it supports open/closed, h2, http1, detailed latencies.

Are you aware of what other stats or features would be provided by nighthawk? What are the NH features (I saw many options) that you think are missing in Bombardier? For grpc it's just a service that mocks a grpc service, or is that something specific on the client too (/cc @JamesNK)?

I couldn't find any binaries in the releases page on GH. How would you handle that for deployment? Does it work on windows too?

[oschaaf]

Facilitating releases for Windows might make this harder than I initially thought. But, while I don't think it will build on Windows out of the box, as Nighthawk is build on Envoy's libraries, and that does build on Windows these days, I think Windows support might be low hanging fruit. I'd better make sure about that before making such proposals though, good point :-)

For convenience we push docker images from CI, which so far has sufficed. There's a backwards compatibility guarantee for the cli and proto inputs.

One of the key things it tries to be pedantic about is request-release timings. In some benchmarks involving low latencies, (unintended) batchy traffic emitted by the load gen client can turn out to be a dominant factor. For example, one of the (many) inputs in the decision to initiate Nighthawk was a comparison between wrk2 and Fortio https://docs.google.com/document/d/10xeQuEjUjdmfFq36kGrxI6v78GKHspqn_Nb-pohKfOo/edit#heading=h.8sjyylhihxed

Bombardier was new to me, so I can't really say much about how goals, features and properties intersect. But here's a few things that seem to not overlap, when looking at the docs and comparing load gen clients based on that:

• Capacity to send bursts of requests
• Capacity to add jitter to request-release timings according to a distributions
• Explicit connection re-use strategies (LRU, MRU)
• Controlling pool queue sizes
• Controlling (h2) streams / connection
• Connection prefetching
• Ability to explicitly run in closed or open loop mode
• Replaying traffic from external sources
• Tracing (e.g. zipkin)
• Histogram stats flushing to external sinks
• One of the gRPC services that comes with Nighthawk accepts a load test configuration, executes it, and returns results. You can specify --nighthawk-service as an argument to the client CLI and it will delegate the load test that way. There's active development on facilitating horizontally scaled testing, where this will be a building block that facilitates it, amongst others.
• Adaptive load controller (e.g for finding a set qps under max latency)
• Capacity for sourcing to-be-replayed traffic (via plugins, e.g file, gRPC).
• For a statistics backend, HdrHistogram is currently used as the primary one, but libcirclhist is already baked in and will probably be exposed as an alternative through a new option, based on https://github.com/envoyproxy/nighthawk/issues/115

[sebastienros]

Being Linux only is not a blocker, just nice to be able to use a local Windows agent too for testing.

How does "Replaying traffic from external sources" work? I couldn't find it in the readme.

[sebastienros]

Maybe that's related to --multi-target-endpoint ?

[oschaaf]

Re: Maybe that's related to --multi-target-endpoint ?

That's for spreading traffic around multiple endpoints;

Re: How does "Replaying traffic from external sources" work?

Today, there's:

--request-source <uri format>
Remote gRPC source that will deliver to-be-replayed traffic. Each
worker will separately connect to this source. For example
grpc://127.0.0.1:8443/.

(The feature is in flux at the moment as it is being extended. I think there's more possible right now but docs/configs are not up yet, e.g. https://github.com/envoyproxy/nighthawk/blob/master/api/client/options.proto mentions RequestSourcePlugins)