carbonapi [BUG] HTTP 200 when backend servers timeout

Describe the bug I am doing a tagValues request from Graphite that takes some time on backend servers. We can see in logs it times-out, but the response carbonapi gives is still 200 ok, with just zero results.

CarbonAPI Version 0.15.4

Logs

Nov 30 09:42:14 defassa carbonapi[19351]: 2021-11-30T09:42:14.988+0100        DEBUG        zipper        single fetch        {"type": "broadcastGroup", "groupName": "root", "query": "expr=technology%3DMPA&expr=name%3D~interface_%28in%7Cout%29_octets&from=1638175333&limit=10000&tag=hostname&until=1638261735", "type": "tagValues", "client": {}}
Nov 30 09:42:14 defassa carbonapi[19351]: 2021-11-30T09:42:14.988+0100        DEBUG        zipper        waiting for a slot        {"type": "broadcastGroup", "groupName": "root", "query": "expr=technology%3DMPA&expr=name%3D~interface_%28in%7Cout%29_octets&from=1638175333&limit=10000&tag=hostname&until=1638261735", "type": "tagValues", "group_name": "root", "backend_name": "clickhouse_cluster"}
Nov 30 09:42:14 defassa carbonapi[19351]: 2021-11-30T09:42:14.988+0100        DEBUG        zipper        got a slot        {"type": "broadcastGroup", "groupName": "root", "query": "expr=technology%3DMPA&expr=name%3D~interface_%28in%7Cout%29_octets&from=1638175333&limit=10000&tag=hostname&until=1638261735", "type": "tagValues", "group_name": "root", "backend_name": "clickhouse_cluster"}
Nov 30 09:42:14 defassa carbonapi[19351]: 2021-11-30T09:42:14.988+0100        DEBUG        zipper        single fetch        {"type": "broadcastGroup", "groupName": "clickhouse_cluster", "query": "expr=technology%3DMPA&expr=name%3D~interface_%28in%7Cout%29_octets&from=1638175333&limit=10000&tag=hostname&until=1638261735", "type": "tagValues", "client": {}}
Nov 30 09:42:14 defassa carbonapi[19351]: 2021-11-30T09:42:14.988+0100        DEBUG        zipper        single fetch        {"type": "broadcastGroup", "groupName": "clickhouse_cluster", "query": "expr=technology%3DMPA&expr=name%3D~interface_%28in%7Cout%29_octets&from=1638175333&limit=10000&tag=hostname&until=1638261735", "type": "tagValues", "client": {}}
Nov 30 09:42:14 defassa carbonapi[19351]: 2021-11-30T09:42:14.988+0100        DEBUG        zipper        waiting for a slot        {"type": "broadcastGroup", "groupName": "clickhouse_cluster", "query": "expr=technology%3DMPA&expr=name%3D~interface_%28in%7Cout%29_octets&from=1638175333&limit=10000&tag=hostname&until=1638261735", "type": "tagValues", "group_name": "clickhouse_cluster", "backend_name": "http://backend-2:9090"}
Nov 30 09:42:14 defassa carbonapi[19351]: 2021-11-30T09:42:14.988+0100        DEBUG        zipper        got a slot        {"type": "broadcastGroup", "groupName": "clickhouse_cluster", "query": "expr=technology%3DMPA&expr=name%3D~interface_%28in%7Cout%29_octets&from=1638175333&limit=10000&tag=hostname&until=1638261735", "type": "tagValues", "group_name": "clickhouse_cluster", "backend_name": "http://backend-2:9090"}
Nov 30 09:42:14 defassa carbonapi[19351]: 2021-11-30T09:42:14.988+0100        DEBUG        zipper        waiting for a slot        {"type": "broadcastGroup", "groupName": "clickhouse_cluster", "query": "expr=technology%3DMPA&expr=name%3D~interface_%28in%7Cout%29_octets&from=1638175333&limit=10000&tag=hostname&until=1638261735", "type": "tagValues", "group_name": "clickhouse_cluster", "backend_name": "http://backend-1:9090"}
Nov 30 09:42:14 defassa carbonapi[19351]: 2021-11-30T09:42:14.988+0100        DEBUG        zipper        got a slot        {"type": "broadcastGroup", "groupName": "clickhouse_cluster", "query": "expr=technology%3DMPA&expr=name%3D~interface_%28in%7Cout%29_octets&from=1638175333&limit=10000&tag=hostname&until=1638261735", "type": "tagValues", "group_name": "clickhouse_cluster", "backend_name": "http://backend-1:9090"}
Nov 30 09:42:14 defassa carbonapi[19351]: 2021-11-30T09:42:14.988+0100        DEBUG        zipper        trying to get slot        {"type": "protoV2Group", "name": "http://backend-2:9090", "type": "tagValues", "carbonapi_uuid": "", "function": "HttpQuery.doRequest", "server": "http://backend-2:9090", "name": "http://backend-2:9090", "uri": "http://backend-2:9090/tags/autoComplete/values?expr=technology%3DMPA&expr=name%3D~interface_%28in%7Cout%29_octets&from=1638175333&limit=10000&tag=hostname&until=1638261735", "name": "http://backend-2:9090"}
Nov 30 09:42:14 defassa carbonapi[19351]: 2021-11-30T09:42:14.988+0100        DEBUG        zipper        got slot for server        {"type": "protoV2Group", "name": "http://backend-2:9090", "type": "tagValues", "carbonapi_uuid": "", "function": "HttpQuery.doRequest", "server": "http://backend-2:9090", "name": "http://backend-2:9090", "uri": "http://backend-2:9090/tags/autoComplete/values?expr=technology%3DMPA&expr=name%3D~interface_%28in%7Cout%29_octets&from=1638175333&limit=10000&tag=hostname&until=1638261735", "name": "http://backend-2:9090"}
Nov 30 09:42:14 defassa carbonapi[19351]: 2021-11-30T09:42:14.989+0100        DEBUG        zipper        trying to get slot        {"type": "protoV2Group", "name": "http://backend-1:9090", "type": "tagValues", "carbonapi_uuid": "", "function": "HttpQuery.doRequest", "server": "http://backend-1:9090", "name": "http://backend-1:9090", "uri": "http://backend-1:9090/tags/autoComplete/values?expr=technology%3DMPA&expr=name%3D~interface_%28in%7Cout%29_octets&from=1638175333&limit=10000&tag=hostname&until=1638261735", "name": "http://backend-1:9090"}
Nov 30 09:42:14 defassa carbonapi[19351]: 2021-11-30T09:42:14.989+0100        DEBUG        zipper        got slot for server        {"type": "protoV2Group", "name": "http://backend-1:9090", "type": "tagValues", "carbonapi_uuid": "", "function": "HttpQuery.doRequest", "server": "http://backend-1:9090", "name": "http://backend-1:9090", "uri": "http://backend-1:9090/tags/autoComplete/values?expr=technology%3DMPA&expr=name%3D~interface_%28in%7Cout%29_octets&from=1638175333&limit=10000&tag=hostname&until=1638261735", "name": "http://backend-1:9090"}
Nov 30 09:42:16 defassa carbonapi[19351]: 2021-11-30T09:42:16.988+0100        DEBUG        zipper        got some responses        {"type": "broadcastGroup", "groupName": "clickhouse_cluster", "query": "expr=technology%3DMPA&expr=name%3D~interface_%28in%7Cout%29_octets&from=1638175333&limit=10000&tag=hostname&until=1638261735", "type": "tagValues", "backends_count": 2, "response_count": 0, "have_errors": true}
Nov 30 09:42:16 defassa carbonapi[19351]: 2021-11-30T09:42:16.989+0100        DEBUG        zipper        got some responses        {"type": "broadcastGroup", "groupName": "root", "query": "expr=technology%3DMPA&expr=name%3D~interface_%28in%7Cout%29_octets&from=1638175333&limit=10000&tag=hostname&until=1638261735", "type": "tagValues", "backends_count": 1, "response_count": 0, "have_errors": true}
Nov 30 09:42:16 defassa carbonapi[19351]: 2021-11-30T09:42:16.989+0100        DEBUG        zipper        error fetching result        {"type": "protoV2Group", "name": "http://backend-2:9090", "type": "tagValues", "carbonapi_uuid": "", "function": "HttpQuery.doRequest", "server": "http://backend-2:9090", "name": "http://backend-2:9090", "uri": "http://backend-2:9090/tags/autoComplete/values?expr=technology%3DMPA&expr=name%3D~interface_%28in%7Cout%29_octets&from=1638175333&limit=10000&tag=hostname&until=1638261735", "error": "Get \"http://backend-2:9090/tags/autoComplete/values?expr=technology%3DMPA&expr=name%3D~interface_%28in%7Cout%29_octets&from=1638175333&limit=10000&tag=hostname&until=1638261735\": context deadline exceeded"}
Nov 30 09:42:16 defassa carbonapi[19351]: 2021-11-30T09:42:16.989+0100        DEBUG        zipper        have errors        {"type": "protoV2Group", "name": "http://backend-2:9090", "type": "tagValues", "carbonapi_uuid": "", "error": "timeout while fetching Response: Get \"http://backend-2:9090/tags/autoComplete/values?expr=technology%3DMPA&expr=name%3D~interface_%28in%7Cout%29_octets&from=1638175333&limit=10000&tag=hostname&until=1638261735\": context deadline exceeded", "errorVerbose": "timeout while fetching Response\nHTTP Code: 504\n\ngithub.com/go-graphite/carbonapi/zipper/types.init\n\t/root/go/src/github.com/go-graphite/carbonapi/zipper/types/errors.go:20\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:6309\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:6286\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:6286\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:6286\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:208\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1371\n\n\nCaused By: Get \"http://backend-2:9090/tags/autoComplete/values?expr=technology%3DMPA&expr=name%3D~interface_%28in%7Cout%29_octets&from=1638175333&limit=10000&tag=hostname&until=1638261735\": context deadline exceeded"}
Nov 30 09:42:16 defassa carbonapi[19351]: 2021-11-30T09:42:16.989+0100        DEBUG        zipper        error fetching result        {"type": "protoV2Group", "name": "http://backend-1:9090", "type": "tagValues", "carbonapi_uuid": "", "function": "HttpQuery.doRequest", "server": "http://backend-1:9090", "name": "http://backend-1:9090", "uri": "http://backend-1:9090/tags/autoComplete/values?expr=technology%3DMPA&expr=name%3D~interface_%28in%7Cout%29_octets&from=1638175333&limit=10000&tag=hostname&until=1638261735", "error": "Get \"http://backend-1:9090/tags/autoComplete/values?expr=technology%3DMPA&expr=name%3D~interface_%28in%7Cout%29_octets&from=1638175333&limit=10000&tag=hostname&until=1638261735\": context deadline exceeded"}
Nov 30 09:42:16 defassa carbonapi[19351]: 2021-11-30T09:42:16.989+0100        DEBUG        zipper        trying to get slot        {"type": "protoV2Group", "name": "http://backend-2:9090", "type": "tagValues", "carbonapi_uuid": "", "function": "HttpQuery.doRequest", "server": "http://backend-2:9090", "name": "http://backend-2:9090", "uri": "http://backend-2:9090/tags/autoComplete/values?expr=technology%3DMPA&expr=name%3D~interface_%28in%7Cout%29_octets&from=1638175333&limit=10000&tag=hostname&until=1638261735", "name": "http://backend-2:9090"}
Nov 30 09:42:16 defassa carbonapi[19351]: 2021-11-30T09:42:16.989+0100        DEBUG        zipper        got slot for server        {"type": "protoV2Group", "name": "http://backend-2:9090", "type": "tagValues", "carbonapi_uuid": "", "function": "HttpQuery.doRequest", "server": "http://backend-2:9090", "name": "http://backend-2:9090", "uri": "http://backend-2:9090/tags/autoComplete/values?expr=technology%3DMPA&expr=name%3D~interface_%28in%7Cout%29_octets&from=1638175333&limit=10000&tag=hostname&until=1638261735", "name": "http://backend-2:9090"}
Nov 30 09:42:16 defassa carbonapi[19351]: 2021-11-30T09:42:16.989+0100        DEBUG        zipper        error fetching result        {"type": "protoV2Group", "name": "http://backend-2:9090", "type": "tagValues", "carbonapi_uuid": "", "function": "HttpQuery.doRequest", "server": "http://backend-2:9090", "name": "http://backend-2:9090", "uri": "http://backend-2:9090/tags/autoComplete/values?expr=technology%3DMPA&expr=name%3D~interface_%28in%7Cout%29_octets&from=1638175333&limit=10000&tag=hostname&until=1638261735", "error": "Get \"http://backend-2:9090/tags/autoComplete/values?expr=technology%3DMPA&expr=name%3D~interface_%28in%7Cout%29_octets&from=1638175333&limit=10000&tag=hostname&until=1638261735\": context deadline exceeded"}
Nov 30 09:42:16 defassa carbonapi[19351]: 2021-11-30T09:42:16.989+0100        DEBUG        zipper        have errors        {"type": "protoV2Group", "name": "http://backend-2:9090", "type": "tagValues", "carbonapi_uuid": "", "error": "timeout while fetching Response: Get \"http://backend-2:9090/tags/autoComplete/values?expr=technology%3DMPA&expr=name%3D~interface_%28in%7Cout%29_octets&from=1638175333&limit=10000&tag=hostname&until=1638261735\": context deadline exceeded", "errorVerbose": "timeout while fetching Response\nHTTP Code: 504\n\ngithub.com/go-graphite/carbonapi/zipper/types.init\n\t/root/go/src/github.com/go-graphite/carbonapi/zipper/types/errors.go:20\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:6309\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:6286\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:6286\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:6286\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:208\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1371\n\n\nCaused By: Get \"http://backend-2:9090/tags/autoComplete/values?expr=technology%3DMPA&expr=name%3D~interface_%28in%7Cout%29_octets&from=1638175333&limit=10000&tag=hostname&until=1638261735\": context deadline exceeded"}
Nov 30 09:42:16 defassa carbonapi[19351]: 2021-11-30T09:42:16.989+0100        DEBUG        zipper        have errors        {"type": "protoV2Group", "name": "http://backend-1:9090", "type": "tagValues", "carbonapi_uuid": "", "error": "timeout while fetching Response: Get \"http://backend-1:9090/tags/autoComplete/values?expr=technology%3DMPA&expr=name%3D~interface_%28in%7Cout%29_octets&from=1638175333&limit=10000&tag=hostname&until=1638261735\": context deadline exceeded", "errorVerbose": "timeout while fetching Response\nHTTP Code: 504\n\ngithub.com/go-graphite/carbonapi/zipper/types.init\n\t/root/go/src/github.com/go-graphite/carbonapi/zipper/types/errors.go:20\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:6309\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:6286\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:6286\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:6286\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:208\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1371\n\n\nCaused By: Get \"http://backend-1:9090/tags/autoComplete/values?expr=technology%3DMPA&expr=name%3D~interface_%28in%7Cout%29_octets&from=1638175333&limit=10000&tag=hostname&until=1638261735\": context deadline exceeded"}
Nov 30 09:42:17 defassa carbonapi[19351]: 2021-11-30T09:42:16.989+0100        DEBUG        zipper        trying to get slot        {"type": "protoV2Group", "name": "http://backend-2:9090", "type": "tagValues", "carbonapi_uuid": "", "function": "HttpQuery.doRequest", "server": "http://backend-2:9090", "name": "http://backend-2:9090", "uri": "http://backend-2:9090/tags/autoComplete/values?expr=technology%3DMPA&expr=name%3D~interface_%28in%7Cout%29_octets&from=1638175333&limit=10000&tag=hostname&until=1638261735", "name": "http://backend-2:9090"}
Nov 30 09:42:17 defassa carbonapi[19351]: 2021-11-30T09:42:16.989+0100        DEBUG        zipper        got slot for server        {"type": "protoV2Group", "name": "http://backend-2:9090", "type": "tagValues", "carbonapi_uuid": "", "function": "HttpQuery.doRequest", "server": "http://backend-2:9090", "name": "http://backend-2:9090", "uri": "http://backend-2:9090/tags/autoComplete/values?expr=technology%3DMPA&expr=name%3D~interface_%28in%7Cout%29_octets&from=1638175333&limit=10000&tag=hostname&until=1638261735", "name": "http://backend-2:9090"}
Nov 30 09:42:17 defassa carbonapi[19351]: 2021-11-30T09:42:16.989+0100        DEBUG        zipper        error fetching result        {"type": "protoV2Group", "name": "http://backend-2:9090", "type": "tagValues", "carbonapi_uuid": "", "function": "HttpQuery.doRequest", "server": "http://backend-2:9090", "name": "http://backend-2:9090", "uri": "http://backend-2:9090/tags/autoComplete/values?expr=technology%3DMPA&expr=name%3D~interface_%28in%7Cout%29_octets&from=1638175333&limit=10000&tag=hostname&until=1638261735", "error": "Get \"http://backend-2:9090/tags/autoComplete/values?expr=technology%3DMPA&expr=name%3D~interface_%28in%7Cout%29_octets&from=1638175333&limit=10000&tag=hostname&until=1638261735\": context deadline exceeded"}
Nov 30 09:42:17 defassa carbonapi[19351]: 2021-11-30T09:42:16.989+0100        DEBUG        zipper        trying to get slot        {"type": "protoV2Group", "name": "http://backend-1:9090", "type": "tagValues", "carbonapi_uuid": "", "function": "HttpQuery.doRequest", "server": "http://backend-1:9090", "name": "http://backend-1:9090", "uri": "http://backend-1:9090/tags/autoComplete/values?expr=technology%3DMPA&expr=name%3D~interface_%28in%7Cout%29_octets&from=1638175333&limit=10000&tag=hostname&until=1638261735", "name": "http://backend-1:9090"}
Nov 30 09:42:17 defassa carbonapi[19351]: 2021-11-30T09:42:16.989+0100        DEBUG        zipper        got slot for server        {"type": "protoV2Group", "name": "http://backend-1:9090", "type": "tagValues", "carbonapi_uuid": "", "function": "HttpQuery.doRequest", "server": "http://backend-1:9090", "name": "http://backend-1:9090", "uri": "http://backend-1:9090/tags/autoComplete/values?expr=technology%3DMPA&expr=name%3D~interface_%28in%7Cout%29_octets&from=1638175333&limit=10000&tag=hostname&until=1638261735", "name": "http://backend-1:9090"}
Nov 30 09:42:17 defassa carbonapi[19351]: 2021-11-30T09:42:16.989+0100        DEBUG        zipper        have errors        {"type": "protoV2Group", "name": "http://backend-2:9090", "type": "tagValues", "carbonapi_uuid": "", "error": "timeout while fetching Response: Get \"http://backend-2:9090/tags/autoComplete/values?expr=technology%3DMPA&expr=name%3D~interface_%28in%7Cout%29_octets&from=1638175333&limit=10000&tag=hostname&until=1638261735\": context deadline exceeded", "errorVerbose": "timeout while fetching Response\nHTTP Code: 504\n\ngithub.com/go-graphite/carbonapi/zipper/types.init\n\t/root/go/src/github.com/go-graphite/carbonapi/zipper/types/errors.go:20\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:6309\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:6286\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:6286\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:6286\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:208\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1371\n\n\nCaused By: Get \"http://backend-2:9090/tags/autoComplete/values?expr=technology%3DMPA&expr=name%3D~interface_%28in%7Cout%29_octets&from=1638175333&limit=10000&tag=hostname&until=1638261735\": context deadline exceeded"}
Nov 30 09:42:17 defassa carbonapi[19351]: 2021-11-30T09:42:16.989+0100        DEBUG        zipper        error fetching result        {"type": "protoV2Group", "name": "http://backend-1:9090", "type": "tagValues", "carbonapi_uuid": "", "function": "HttpQuery.doRequest", "server": "http://backend-1:9090", "name": "http://backend-1:9090", "uri": "http://backend-1:9090/tags/autoComplete/values?expr=technology%3DMPA&expr=name%3D~interface_%28in%7Cout%29_octets&from=1638175333&limit=10000&tag=hostname&until=1638261735", "error": "Get \"http://backend-1:9090/tags/autoComplete/values?expr=technology%3DMPA&expr=name%3D~interface_%28in%7Cout%29_octets&from=1638175333&limit=10000&tag=hostname&until=1638261735\": context deadline exceeded"}
Nov 30 09:42:17 defassa carbonapi[19351]: 2021-11-30T09:42:16.989+0100        DEBUG        zipper        have errors        {"type": "protoV2Group", "name": "http://backend-1:9090", "type": "tagValues", "carbonapi_uuid": "", "error": "timeout while fetching Response: Get \"http://backend-1:9090/tags/autoComplete/values?expr=technology%3DMPA&expr=name%3D~interface_%28in%7Cout%29_octets&from=1638175333&limit=10000&tag=hostname&until=1638261735\": context deadline exceeded", "errorVerbose": "timeout while fetching Response\nHTTP Code: 504\n\ngithub.com/go-graphite/carbonapi/zipper/types.init\n\t/root/go/src/github.com/go-graphite/carbonapi/zipper/types/errors.go:20\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:6309\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:6286\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:6286\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:6286\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:208\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1371\n\n\nCaused By: Get \"http://backend-1:9090/tags/autoComplete/values?expr=technology%3DMPA&expr=name%3D~interface_%28in%7Cout%29_octets&from=1638175333&limit=10000&tag=hostname&until=1638261735\": context deadline exceeded"}
Nov 30 09:42:17 defassa carbonapi[19351]: 2021-11-30T09:42:16.989+0100        DEBUG        zipper        trying to get slot        {"type": "protoV2Group", "name": "http://backend-1:9090", "type": "tagValues", "carbonapi_uuid": "", "function": "HttpQuery.doRequest", "server": "http://backend-1:9090", "name": "http://backend-1:9090", "uri": "http://backend-1:9090/tags/autoComplete/values?expr=technology%3DMPA&expr=name%3D~interface_%28in%7Cout%29_octets&from=1638175333&limit=10000&tag=hostname&until=1638261735", "name": "http://backend-1:9090"}
Nov 30 09:42:17 defassa carbonapi[19351]: 2021-11-30T09:42:16.989+0100        DEBUG        zipper        got slot for server        {"type": "protoV2Group", "name": "http://backend-1:9090", "type": "tagValues", "carbonapi_uuid": "", "function": "HttpQuery.doRequest", "server": "http://backend-1:9090", "name": "http://backend-1:9090", "uri": "http://backend-1:9090/tags/autoComplete/values?expr=technology%3DMPA&expr=name%3D~interface_%28in%7Cout%29_octets&from=1638175333&limit=10000&tag=hostname&until=1638261735", "name": "http://backend-1:9090"}
Nov 30 09:42:17 defassa carbonapi[19351]: 2021-11-30T09:42:16.989+0100        DEBUG        zipper        error fetching result        {"type": "protoV2Group", "name": "http://backend-1:9090", "type": "tagValues", "carbonapi_uuid": "", "function": "HttpQuery.doRequest", "server": "http://backend-1:9090", "name": "http://backend-1:9090", "uri": "http://backend-1:9090/tags/autoComplete/values?expr=technology%3DMPA&expr=name%3D~interface_%28in%7Cout%29_octets&from=1638175333&limit=10000&tag=hostname&until=1638261735", "error": "Get \"http://backend-1:9090/tags/autoComplete/values?expr=technology%3DMPA&expr=name%3D~interface_%28in%7Cout%29_octets&from=1638175333&limit=10000&tag=hostname&until=1638261735\": context deadline exceeded"}
Nov 30 09:42:17 defassa carbonapi[19351]: 2021-11-30T09:42:16.989+0100        DEBUG        zipper        have errors        {"type": "protoV2Group", "name": "http://backend-1:9090", "type": "tagValues", "carbonapi_uuid": "", "error": "timeout while fetching Response: Get \"http://backend-1:9090/tags/autoComplete/values?expr=technology%3DMPA&expr=name%3D~interface_%28in%7Cout%29_octets&from=1638175333&limit=10000&tag=hostname&until=1638261735\": context deadline exceeded", "errorVerbose": "timeout while fetching Response\nHTTP Code: 504\n\ngithub.com/go-graphite/carbonapi/zipper/types.init\n\t/root/go/src/github.com/go-graphite/carbonapi/zipper/types/errors.go:20\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:6309\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:6286\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:6286\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:6286\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:208\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1371\n\n\nCaused By: Get \"http://backend-1:9090/tags/autoComplete/values?expr=technology%3DMPA&expr=name%3D~interface_%28in%7Cout%29_octets&from=1638175333&limit=10000&tag=hostname&until=1638261735\": context deadline exceeded"}
Nov 30 09:42:17 defassa carbonapi[19351]: 2021-11-30T09:42:16.989+0100        INFO        access        request served        {"data": {"handler":"tags","carbonapi_uuid":"9163d155-e733-4f01-b5f4-06dbf419aa53","url":"/tags/autoComplete/values","peer_ip":"x.x.x.x","host":"localhost:8081","runtime":2.001040137,"http_code":200,"uri":"/tags/autoComplete/values?expr=technology%3DMPA&expr=name%3D~interface_(in%7Cout)_octets&tag=hostname&limit=10000&from=1638175333&until=1638261735","from_cache":false,"used_backend_cache":false,"request_headers":{"X-Grafana-Org-Id":"1"}}}

CarbonAPI Configuration: (Don't know if applicable in this case)

Simplified query (if applicable) tag_values(hostname, technology=MPA, name=~interface_(in|out)_octets)

Backend metric retention and aggregation schemas Please provide backend's schema (most important thing - if query cross retention period or not), aggregation function, xFilesFactor (if applicable).

Backend response (if possible) Backends respond normally (as expected), but takes longer than configured timeout.

Dec 01 '21 11:12 Hipska

Hi @Hipska

IMO carbonapi configuration is 100% applicable in this case. Could you please share it?

Dec 01 '21 14:12 deniszh

Oh, okay then:

# Need to be URL, http or https
# This url specifies the backend or a loadbalancer
#
# If you are using carbonzipper you should set it to
# zipper's url
#
# If you are using plain go-carbon or graphite-clickhouse
# you should set it to URL of go-carbon's carbonserver module
# or graphite-clickhouse's http url.
# Listen address, should always include hostname or ip address and a port.
listen: "0.0.0.0:8081"
# Allow extra charsets in metric names. By default only "Latin" is allowed
# Please note that each unicodeRangeTables will slow down metric parsing a bit
#   For list of supported tables, see: https://golang.org/src/unicode/tables.go?#L3437
#   Special name "all" reserved to append all tables that's currently supported by Go
#unicodeRangeTables:
#   - "Latin"
#   - "Cyrillic"
#   - "Hiragana"
#   - "Katakana"
#   - "Han"
##   - "all"
# Controls headers that would be passed to the backend
headersToPass:
  - "X-Dashboard-Id"
  - "X-Grafana-Org-Id"
  - "X-Panel-Id"
headersToLog:
  - "X-Dashboard-Id"
  - "X-Grafana-Org-Id"
  - "X-Panel-Id"
# Max concurrent requests to CarbonZipper
concurency: 1000
cache:
   # Type of caching. Valid: "mem", "memcache", "null"
   type: "mem"
   # Cache limit in megabytes
   size_mb: 0
   # Default cache timeout value. Identical to DEFAULT_CACHE_DURATION in graphite-web.
   defaultTimeoutSec: 60
   # Only used by memcache type of cache. List of memcache servers.
   memcachedServers:
      # - "127.0.0.1:1234"
      # - "127.0.0.2:1235"

# Amount of CPUs to use. 0 - unlimited
cpus: 0
# Timezone, default - local
tz: ""
# If 'true', carbonapi will send requests as is, with globs and braces
# Otherwise for each request it will generate /metrics/find and then /render
# individual metrics.
# true --- faster, but will cause carbonzipper to consume much more RAM.
#
# For some backends (e.x. graphite-clickhouse) this *MUST* be set to true in order
# to get reasonable performance
#
# For go-carbon --- it depends on how you use it.
sendGlobsAsIs: true
# If sendGlobsAsIs is set and resulting response will be larger than maxBatchSize
# carbonapi will revert to old behavir. This allows you to use benifits of passing
# globs as is but keep memory usage in sane limits.
#
# For go-carbon you might want it to keep in some reasonable limits, 100 is good "safe" defaults
#
# For some backends (e.x. graphite-clickhouse) you might want to set it to some insanly high value, like 100000
maxBatchSize: 100000
graphite:
    # Host:port where to send internal metrics
    # Empty = disabled
    host: "x.x.x.x:2003"
    interval: "60s"
    prefix: "carbon.api"
    # rules on how to construct metric name. For now only {prefix} and {fqdn} is supported.
    # {prefix} will be replaced with the content of {prefix}
    # {fqdn} will be repalced with fqdn
    pattern: "{prefix}.{fqdn}"
# Maximium idle connections to carbonzipper
idleConnections: 10
pidFile: ""
# See https://github.com/go-graphite/carbonzipper/blob/master/example.conf#L70-L108 for format explanation
upstreams:
    # Number of 100ms buckets to track request distribution in. Used to build
    # 'carbon.zipper.hostname.requests_in_0ms_to_100ms' metric and friends.
    # Requests beyond the last bucket are logged as slow (default of 10 implies
    # "slow" is >1 second).
    # The last bucket is _not_ called 'requests_in_Xms_to_inf' on purpose, so
    # we can change our minds about how many buckets we want to have and have
    # their names remain consistent.
    buckets: 10

    timeouts:
        # Maximum backend request time for find requests.
        find: "20s"
        # Maximum backend request time for render requests. This is total one and doesn't take into account in-flight requests
        render: "50s"
        # Timeout to connect to the server
        connect: "200ms"

    # Number of concurrent requests to any given backend - default is no limit.
    # If set, you likely want >= MaxIdleConnsPerHost
    concurrencyLimitPerServer: 0

    # Configures how often keep alive packets will be sent out
    keepAliveInterval: "30s"

    # Control http.MaxIdleConnsPerHost. Large values can lead to more idle
    # connections on the backend servers which may bump into limits; tune with care.
    maxIdleConnsPerHost: 100

    backendsv2:
        backends:
          -
            groupName: "clickhouse_cluster"
            # supported:
            #    carbonapi_v2_pb - carbonapi 0.11 or earlier version of protocol.
            #    carbonapi_v3_pb - new protocol, http interface (native)
            #    carbonapi_v3_grpc - new protocol, gRPC interface (native)
            #    protobuf, pb, pb3 - same as carbonapi_v2_pb
            #    msgpack - protocol used by graphite-web 1.1 and metrictank
            #    auto - carbonapi will do it's best to guess if it's carbonapi_v3_pb or carbonapi_v2_pb
            #
            #  non-native protocols will be internally converted to new protocol, which will increase memory consumption
            protocol: "carbonapi_v2_pb"
            # supported:
            #    "broadcast" - send request to all backends in group and merge responses. This was default behavior for carbonapi 0.11 or earlier
            #    "roundrobin" - send request to one backend.
            #    "all - same as "broadcast"
            #    "rr" - same as "roundrobin"
            lbMethod: "broadcast"
            # amount of retries in case of unsuccessful request
            maxTries: 3
            # amount of metrics per fetch request. Default: 0 - unlimited. If not specified, global will be used
            maxBatchSize: 0
            # interval for keep-alive http packets. If not specified, global will be used
            keepAliveInterval: "10s"
            # override for global concurrencyLimit.
            concurrencyLimit: 0
            # override for global maxIdleConnsPerHost
            maxIdleConnsPerHost: 1000
            # per-group timeout override. If not specified, global will be used.
            # Please note that ONLY min(global, local) will be used.
            timeouts:
                # Maximum backend request time for find requests.
                find: "20s"
                # Maximum backend request time for render requests. This is total one and doesn't take into account in-flight requests.
                render: "50s"
                # Timeout to connect to the server
                connect: "200ms"
            servers:
                - "http://backend-1:9090"
                - "http://backend-2:9090"

    # carbonsearch is not used if empty
    carbonsearch:
        # Instance of carbonsearch backend
        # backend: "http://127.0.0.1:8070"
        # carbonsearch prefix to reserve/register
        prefix: "virt.v1.*"

    # Enable compatibility with graphite-web 0.9
    # This will affect graphite-web 1.0+ with multiple cluster_servers
    # Default: disabled
    graphite09compat: false

# If not zero, enabled cache for find requests
# This parameter controls when it will expire (in seconds)
# Default: 600 (10 minutes)
expireDelaySec: 10
# Uncomment this to get the behavior of graphite-web as proposed in https://github.com/graphite-project/graphite-web/pull/2239
# Beware this will make darkbackground graphs less readable
#defaultColors:
#      "red": "ff0000"
#      "green": "00ff00"
#      "blue": "#0000ff"
#      "darkred": "#c80032"
#      "darkgreen": "00c800"
#      "darkblue": "002173"
logger:
    - logger: ""
      file: "stderr"
      level: "debug"
      encoding: "console"
      encodingTime: "iso8601"
      encodingDuration: "seconds"
    - logger: ""
      file: "/var/log/carbonapi/carbonapi.log"
      level: "info"
      encoding: "json"

Dec 01 '21 14:12 Hipska

@deniszh any update?

Dec 09 '22 08:12 Hipska

I think 200 OK can still be returned as a result if one backend replies.

Also https://github.com/go-graphite/carbonapi/blob/main/doc/configuration.md#notfoundstatuscode

by default for not found status code is set to 200 (for historical reasons) you can try to override that in the config.

Dec 09 '22 08:12 Civil

Did one reply in time? I could not find that in the logs, they are very hard to read.

And is that setting the one to set also for this case? (timeout and doing a call to /tags/autoComplete/values)

Dec 09 '22 09:12 Hipska

I planned add error handling to metrics find/tags autocomplete. Now always return 200 code (if error - with empty results).

Jun 19 '23 13:06 msaf1980