[FR] Make upstream auto-configuration timeout configurable

[jhass]

There's even a very old TODO for this :)

https://github.com/go-graphite/carbonapi/blob/556583e7134405aeb17692875ba10b955b2d48ab/zipper/protocols/auto/auto_group.go#L88-L89

[Civil]

Well, this is timeout only for querying /_internal/capabilities handler to determine if backend supports carbonapi_v3_pb. In both graphite-clickhouse and go-carbon those are just returning static json back, which should be fast unless you have backend almost across the globe or over very slow internet connection. And as I'm not aware about such installations, that was never a priority to fix (as that would require some very specific configuration options only for auto backends).

[jhass]

I have this trigger sometimes when my backup scripts freeze the VM briefly to snapshot its disk :)

Well or maybe the other place using the same error message, not 100% sure to be honest, but I didn't manage to workaround it with any of the existing timeout options.

[Felixoid]

From my personal experience, auto is useful only when you migrate the backend from one to another protocol. But even then, it's useless, because you have to restart carbonapi, since it doesn't retrigger the protocol.

I'd suggest using a pre-defined protocol, it's less error-prone.

[jhass]

It might be worth to give better examples in the readme then :) Or even a full-fledged and commented example configuration that packages can install.

[Civil]

Well or maybe the other place using the same error message, not 100% sure to be honest, but I didn't manage to workaround it with any of the existing timeout options.

Could you please show the logs that you had?

[jhass]

I'm seeing things like this then:

2020-10-16T02:59:12.999Z  WARN    main    timeout waiting for more responses      {"handler": "zipper", "handler": "multiGet", "uri": "/render/?format=protobuf&from=1602816850&target=mail_aeshna_de.deferred.postfix_queue.length&until=1602817150", "timeouted_servers": ["http://127.0.0.1:8080"], "answers_from_servers": []}
2020-10-16T02:59:12.999Z  ERROR   render  render error occurred while fetching data       {"errors": [{"error": "No responses fetched from upstream"}]}