VictoriaMetrics
VictoriaMetrics copied to clipboard
VictoriaMetrics
vmagent doesn't follow redirects without stream_parse
Describe the bug vmagent doesn't follow redirects without setting stream_parse: true for details see slack thread: https://victoriametrics.slack.com/archives/CGZF1H6L9/p1639404018163100
Expected behavior it should follow redirects in a way that is similar to prometheus
VM agent config
global:
scrape_interval: 1m0s
scrape_configs:
- job_name: emr
follow_redirects: true
relabel_configs:
- source_labels: [__meta_ec2_private_ip]
target_label: __metrics_path__
replacement: /bi-emr-reverse-logs/$1:4040/metrics/executors/prometheus/
- source_labels: [__meta_ec2_tag_Name]
target_label: instance
- target_label: __address__
replacement: bo.XX.com
ec2_sd_configs:
- region: us-east-1
access_key: XXX
secret_key: <secret>
filters:
- name: tag:aws:elasticmapreduce:instance-group-role
values:
- MASTER
- name: tag:Name
values:
- airflow-shared-persistent-emr-6.3.0
Version tried v1.7.0 and v1.68.0 and latest docker version
**Logs**
`unexpected status code returned when scraping "http://bo.XXX.com:80/bi-emr-reverse-logs/10.207.109.205:4040/metrics/executors/prometheus/": 301; expecting 200; response body: "<html>\r\n<head><title>301 Moved Permanently</title></head>\r\n<body>\r\n<center><h1>301 Moved Permanently</h1></center>\r\n<hr><center>openresty/1.19.3.1</center>\r\n</body>\r\n</html>\r\n"`
**Workaround**
the issue was resolved by adding stream_parse: true
vmagent was following only a single redirect up to v1.70.0 . It should follow up to 5 redirects when built from the commit a3adf24527336a233cacd5341fca42289c147413 . @vbichov , could you build vmagent from this commit according to these docs and verify whether this allows scraping targets without stream_parse: true mode?
Note that vmagent expects that redirects are performed to the original hostname when stream_parse: true isn't set in scrape config.
vmagent should follow for up to 5 redirects starting from v1.71.0. @vbichov , could you check whether this helps resolving the issue?
Hello,
For information, I still have the issue with vmagent v1.79.5...
2022-12-14T09:46:18.261Z warn VictoriaMetrics/lib/promscrape/scrapework.go:385 cannot scrape "http://xxxxxxx:80/snmp?module=icx&target=xxxxxxx" (job "Switches-foundry", labels {datacenter="XX",env="XX",instance="XX",job="Switches-foundry",monitor="prometheus04",site="XX"}): error when scraping "http://xxxxxxx:80/snmp?module=icx&target=xxxxxxx": too many redirects
2022-12-14T09:46:18.280Z warn VictoriaMetrics/lib/promscrape/scrapework.go:385 cannot scrape "http://xxxxxxx:80/probe?module=icmp&target=xxxxxxx" (job "Switches-ping", labels {datacenter="XX",env="XX",instance="XX",job="Switches-ping",monitor="prometheus04",site="XX"}): error when scraping "http://xxxxxxx:80/probe?module=icmp&target=xxxxxxx": too many redirects
2022-12-14T09:46:18.379Z warn VictoriaMetrics/lib/promscrape/scrapework.go:385 cannot scrape "http://xxxxxxx:80/probe?module=icmp&target=xxxxxxx" (job "Routers-ping", labels {datacenter="XX",env="XX",instance="XX",job="Routers-ping",monitor="prometheus04",site="XX"}): error when scraping "http://xxxxxxx:80/probe?module=icmp&target=xxxxxxx": too many redirects
@earendilfr do you know how many redirects are there? It could be checked via browser...
@earendilfr do you know how many redirects are there? It could be checked via browser...
@hagen1778 : sorry, I have changed of jobs so hard to me to have this information but, if I remember correctly, I have only one redirect because, in this case, it's was just a redirect from http to https on the same server.
@earendilfr do you know how many redirects are there? It could be checked via browser...
@hagen1778 : sorry, I have changed of jobs so hard to me to have this information but, if I remember correctly, I have only one redirect because, in this case, it's was just a redirect from http to https on the same server.
hi, have you solved this？ I am facing the same problem that it shows too many redirect while directing from http to https.
we just ran into this same issue: a single hop redirect http -> https fails with too many redirects
we enabled streamParse for all of vmagent via arguments, but i don't feel we should have to do this
and after enabling streamParse we see some new skipping duplicate scrape target with identical labels errors for some of our targets; does streamParse change how relabelling works?
we just ran into this same issue: a single hop redirect http -> https fails with too many redirects
Is it possible to compile docker-compose env where this issue can be reproduced? This would simplify debug from our side.
and after enabling streamParse we see some new skipping duplicate scrape target with identical labels errors for some of our targets; does streamParse change how relabelling works?
Not that I'm aware of. Have you checked https://docs.victoriametrics.com/vmagent.html#troubleshooting ?
I just turned off streamParse and confirmed we're still facing this problem. Here's what the redirect looks like:
The metrics target is cockroachDB. I'll see if I can reproduce in docker-compose or even better, narrow down the problem in the code.
After reading some code, perhaps this behavior is unfortunate but expected. https://github.com/VictoriaMetrics/VictoriaMetrics/blob/fb9b8f6b1b51d3b0aa235d8fc2a7d4ec608dea73/lib/promscrape/client.go#L291
I expect that http->https qualifies as changing the host, but fasthttp is reusing the existing connection to port 80, so of course this will be a redirect loop as the redirect was not followed at all.
I believe it will be possible to fix this case, but @valyala may not love it
If we detect this case on redirect in promscrape client.go and create new c.hc = &fasthttp.HostClient{... with IsTLS now set appropriately, it will work thereafter.
Here's a PR (the code is awful! it's just to communicate the idea) that shows it is possible to fix this behavior https://github.com/VictoriaMetrics/VictoriaMetrics/pull/4897
I believe, the issue would be fixed if we completely migrate vmagent to native HTTP lib. @zekker6 I recall you was working on that some time ago... wdyt?
@hagen1778 When I've been testing this native HTTP lib was consuming more memory than fasthttp, and we decided to keep it as is in order to avoid increasing resource usage during scrapes.
It seems to me like it will be better to add a logic to re-create fasthttp.HostClient in case of redirects as it will create much smaller memory footprint than switching to native HTTP lib.
Agreed, moving away from fasthttp would probably hurt performance in a way that might surprise users. This change gets it a little closer to the normal behavior expected from the native client.
Note that in my example here I'm only handling http->https redirects but "different hostname" redirects could also be handled with the same approach.
If you'd like me to do any more work down this road, or clean up this code a bit, just let me know! I factored the creation of the fasthttp client out of the new method, and could do the same for the streamparse client for consistency (not that there's any reason to recreate that one at runtime).
I ran into this with federation targets. This specifically seems to be a problem because it looks like prometheus doesn't let you write https:// in the target, but VM defaults to http and fails on the redirect.
I think this means that you can't write a completely compatible configuration that works on both (this is probably only an issue while transitioning)
I can confirm that at least vmagent v1.93.4 does follow redirects only when stream_parse is true, no matter if follow_redirects is set to true or false. I. e. follow_redirects has no effect at all.
FYI, the commit bc7cf4950b5ff1072a33fe58c6bae1184d01068b should fix following redirects in stream parsing mode when the redirect is performed to different hostname. This commit will be included in the next release. In the mean time it is possible to build vmagent from this commit according to these docs and verify whether follow_redirects: true option in scrape_configs works as intended.