Allow proxy config per request

[Shamunr]

When setting up a request to be sent, being able to set a proxy for every request would be very helpful for my use case. We need to test our proxy code to ensure it's stable and being able to create a different proxy for each VU would be great.

Currently it only suports a single HTTP proxy.

[na--]

Thanks for adding this issue! Unfortunately, this is going to need further evaluation in order to be implemented without introducing any performance regressions.

My concern regarding performance stems from the fact that the proxy is configured in the http.Transport that each VU has. We cannot by default initialize a new transport for each individual request, since it is also responsible for things like http2 multiplexing and the connection reuse between requests to the same host. Even initializing a new transport only for requests that have this new proxy parameter specified is probably not ideal - neither from an UX standpoint (most people probably want a single proxy for a group of requests), nor because we won't be able to reuse the connection to the proxy.

This issue, together with https://github.com/loadimpact/k6/issues/970 (h2c connections) and https://github.com/loadimpact/k6/issues/936 (forced http1.1 connections) makes me think that we need a way to create new custom HTTP transports programmatically from the JavaScript code. Maybe a new method in the http module that would return a new pre-configured HTTP client with a custom transport.

Something like this:

import http from "k6/http";

let myClient = http.NewClient({
  proxy: "https://myproxy",
  forceHTTP1: true,
  // ... other options like the h2c ones, tls versions, etc.
});

export default function () {
    myClient.get("https://httpbin.org/");
    // ...
}

This won't stop you from using a different proxy for each individual request, as you want for your use case, though it will make it slightly more verbose than just specifying the proxy as a parameter to the http.request() call. But I'd prefer this approach, since it would make the tradeoffs clearer and it's not muddling the abstractions.

Some other things that need to be considered:

• Are there any other options that it makes sense to put in the custom clients? There are a lot of HTTP transport things that are configured at the VU level at the moment. I think it makes sense to allow the configuration of a lot of them for the custom transports... Maybe even deprecate some of them on the global level - do we really need a global k6 option for specifying the min and max TLS versions and the cipher suite?
• Once we have a flexible transport configuration, it might make sense to also support a global configuration for the default transport that reuses the same format and code. Basically, instead of having those tlsVersion, tlsCipherSuites, etc. options, we should have a single httpClient (or something) option that configures the default VU transport in the same way a custom transport could be configured.
• We probably should be very strict with the validation for the options of these transports, since there would be some mutually exclusive ones...
• We need to evaluate if there would be any conflicts with https://github.com/loadimpact/k6/issues/761 and https://github.com/loadimpact/k6/issues/884, and also make sure that we don't worsen the configuration mess (https://github.com/loadimpact/k6/issues/883) we currently have. I actually think that the refactoring necessary for this issue could also be used to slightly improve the overall configuration situation, but we'll see...

[mstoykov]

Hm ... I think this will work and it is somewhat (a lot IMO) similar to how golang stdlib does things.

I would prefer, though, to first somewhat fix the configuration mess and than add even more ... configuration.

I also think that the part with changing the default clients practically implements #761. I myself too think that having cli option for everything is overkill and probably not very useful (as I have said before :)).

[na--]

Another somewhat connected issue that could be more easily implemented if we had support for creating custom HTTP clients: https://github.com/loadimpact/k6/issues/857

[na--]

@MStoykov, this would need further evaluation, but I'm not sure the best places for things like common headers and such is the HTTP client. The reason for that is that groups of HTTP requests with completely different headers could still share the same underlying HTTP transport and reuse the same previously opened connections. It's basically also muddling the abstractions, just in the opposite way of having the proxy be configured per-request.

Added to that, I think the use cases for the custom transports I listed or linked above are different than the use cases for custom HTTP request headers, cookies, metric tags, response types, timeouts, etc. I imagine that I'd like to configure those much more often than I'd like a new transport, for situations like that:

let loginData = http.post("https://myapp.com/login")
group("logged in data", doUserStuffOnTheWebsiteCallback, {
  http: {headers: {
    "Authorization": loginData.json().blah,
  }},
  tags: {myCustomMetricTag: "loggedInStuff"},
})

In a use case like that, there's no need to use a different transport, it would actually be somewhat counter-productive.

[baralraj]

How to pass the proxies details with http.post in k6?

I am having issue due to the proxy and I couldn't get the access_token. I am able to get the token for the same with python code when I pass the proxy details but similar error without proxy, So I figure it out the issue is due to the proxy but how do I pass proxy in k6?

NewClient doesn't exist on http and I got error Object has no member 'NewClient'

import http from "k6/http";

export function getAccessToken(clientId, clientSecret, grant_type, auth_url) {
  const data = {
    client_id: clientId,
    client_secret: clientSecret,
    grant_type: grant_type,
  };

  // let myClient = http.NewClient({
  //   proxy: {
  //     http: "http://proxy...",
  //     https: "http://proxy...",
  //   },
  // });

  let res = http.post(auth_url, data, {
    headers: {
      "Content-Type'": "application/x-www-form-urlencoded",
    },
  });

  return res;
}

[na--]

@baralraj, NewClient from my previous comment is just a proposal for how to implement this issue in the future. The issue is still open, this function doesn't exist in k6 yet.

The only way to currently specify a HTTP(s) proxy is globally, one proxy for the whole test run, via the semi-standard HTTP_PROXY, HTTPS_PROXY and NO_PROXY environment variables that a lot of tools support.

[baralraj]

na--

Thank you, do you have any example of how we can set the proxy globally? If yes, can you provide the reference link of that?

I already tried something like this set "HTTP_PROXY=localhost:8888" && k6 run script.js

[na--]

How to set the environment variable depends on the OS, in linux and mac you can just use

HTTP_PROXY=localhost:8888 k6 run script.js

or

export HTTP_PROXY=localhost:8888 
k6 run script.js

In windows I think you can use set instead of export, but here's something I found with examples for all OSes. In any case, please don't ask support questions in unrelated github issues, use the community forum at https://community.k6.io/ instead

[SYM01]

I am facing a similar issue, but no solution had been found.

So I developed a new k6 extension to resolve my own issue.

Example:

import http from 'k6/http';
import proxy from 'k6/x/proxy';

const YOUR_PROXY = 'http://user:[email protected]'

export default function () {
  proxy.setProxy(YOUR_PROXY)

  const resp = http.get('http://httpbin.test.k6.io/get')

  proxy.clearProxy()
}