VictoriaMetrics icon indicating copy to clipboard operation
VictoriaMetrics copied to clipboard
verified publisher icon VictoriaMetrics

VMAgent not sending in VM remote-write and getting error connecting to VM Server with 'response body="\n";' due to Basic Auth

Open OmegaZero opened this issue 2 years ago • 1 comments

Describe the bug

VMAgent executing against VM single node server is

  1. Not remote-writing with VM remote-write and dropping down to Prometheus remote write VictoriaMetrics/app/vmagent/remotewrite/client.go:169 the remote storage at "1:secret-url" doesn't support VictoriaMetrics remote write protocol. Switching to Prometheus remote write protocol.

  2. Getting an error connecting to the VM remote write url (?) VictoriaMetrics/app/vmagent/remotewrite/client.go:465 unexpected status code received after sending a block with size 92156 bytes to "1:secret-url" during retry #3: 401; response body="\n"; re-sending the block in 8.740 seconds I assume it's not authenticating correctly but I'm unsure why

To Reproduce

  • This is on Windows

  • I use windows_exporter with all default settings. I have confirmed that it works fine and I can use Prometheus to scrape it (if not using vmagent)

  • Prometheus.yml:

global:
  scrape_interval: 5s 
  scrape_timeout: 3s 

scrape_configs:
  - job_name: 'sql'
    static_configs:
      - targets: ['localhost:9182']

(Don't worry about the fact that I'm sharing the password, this is just dev and I'll change it later. Including it in case that has something to do with it)

  • I have a VM server running with the following:
victoria-metrics-windows-amd64-prod.exe -retentionPeriod=15 -httpAuth.username=grafana -httpAuth.password=oWsONTEalEtUdGeROXYglEmAKyhwwB
  • I have VMAgent which I run with:
vmagent-windows-amd64-prod.exe -promscrape.config=C:\Monitoring\vmagent\prometheus.yml -httpAuth.username=grafana -httpAuth.password=oWsONTEalEtUdGeROXYglEmAKyhwwB -remoteWrite.url=http://localhost:8428/api/v1/write

Version

victoria-metrics-20240201-133030-tags-v1.97.1-0-g5b065441c vmagent-20240201-133043-tags-v1.97.1-0-g5b065441c

Logs

2024-02-08T21:08:46.505Z        info    VictoriaMetrics/lib/logger/flag.go:12   build version: vmagent-20240201-133043-tags-v1.97.1-0-g5b065441c
2024-02-08T21:08:46.505Z        info    VictoriaMetrics/lib/logger/flag.go:13   command-line flags
2024-02-08T21:08:46.506Z        info    VictoriaMetrics/lib/logger/flag.go:20     -httpAuth.password="secret"
2024-02-08T21:08:46.506Z        info    VictoriaMetrics/lib/logger/flag.go:20     -httpAuth.username="grafana"
2024-02-08T21:08:46.508Z        info    VictoriaMetrics/lib/logger/flag.go:20     -promscrape.config="C:\\Monitoring\\vmagent\\prometheus.yml"
2024-02-08T21:08:46.508Z        info    VictoriaMetrics/lib/logger/flag.go:20     -remoteWrite.url="secret"
2024-02-08T21:08:46.508Z        info    VictoriaMetrics/app/vmagent/main.go:122 starting vmagent at ":8429"...
2024-02-08T21:08:46.509Z        info    VictoriaMetrics/lib/memory/memory.go:42 limiting caches to 5153597030 bytes, leaving 3435731354 bytes to the OS according to -memory.allowedPercent=60
2024-02-08T21:08:46.511Z        info    VictoriaMetrics/lib/persistentqueue/fastqueue.go:66     opened fast persistent queue at "vmagent-remotewrite-data\\persistent-queue\\1_005001D6AF6189A1" with maxInmemoryBlocks=400, it contains 2461848 pending bytes
2024-02-08T21:08:46.528Z        info    VictoriaMetrics/app/vmagent/remotewrite/client.go:169   the remote storage at "1:secret-url" doesn't support VictoriaMetrics remote write protocol. Switching to Prometheus remote write protocol. See https://docs.victoriametrics.com/vmagent.html#victoriametrics-remote-write-protocol
2024-02-08T21:08:46.530Z        info    VictoriaMetrics/app/vmagent/remotewrite/client.go:206   initialized client for -remoteWrite.url="1:secret-url"
2024-02-08T21:08:46.532Z        info    VictoriaMetrics/app/vmagent/main.go:148 started vmagent in 0.024 seconds
2024-02-08T21:08:46.534Z        error   VictoriaMetrics/app/vmagent/remotewrite/client.go:465   unexpected status code received after sending a block with size 92156 bytes to "1:secret-url" during retry #1: 401; response body="\n"; re-sending the block in 2.185 seconds
2024-02-08T21:08:46.538Z        error   VictoriaMetrics/app/vmagent/remotewrite/client.go:465   unexpected status code received after sending a block with size 9853 bytes to "1:secret-url" during retry #1: 401; response body="\n"; re-sending the block in 2.090 seconds
2024-02-08T21:08:46.542Z        info    VictoriaMetrics/lib/promscrape/scraper.go:113   reading scrape configs from "C:\\Monitoring\\vmagent\\prometheus.yml"
2024-02-08T21:08:46.542Z        info    VictoriaMetrics/lib/httpserver/httpserver.go:101        starting http server at http://127.0.0.1:8429/
2024-02-08T21:08:46.548Z        info    VictoriaMetrics/lib/httpserver/httpserver.go:102        pprof handlers are exposed at http://127.0.0.1:8429/debug/pprof/
2024-02-08T21:08:46.548Z        info    VictoriaMetrics/lib/promscrape/config.go:140    starting service discovery routines...
2024-02-08T21:08:46.550Z        info    VictoriaMetrics/lib/promscrape/config.go:146    started 1 service discovery routines in 0.002 seconds
2024-02-08T21:08:46.544Z        error   VictoriaMetrics/app/vmagent/remotewrite/client.go:465   unexpected status code received after sending a block with size 9854 bytes to "1:secret-url" during retry #1: 401; response body="\n"; re-sending the block in 2.089 seconds
2024-02-08T21:08:46.544Z        error   VictoriaMetrics/app/vmagent/remotewrite/client.go:465   unexpected status code received after sending a block with size 92156 bytes to "1:secret-url" during retry #1: 401; response body="\n"; re-sending the block in 2.054 seconds
2024-02-08T21:08:46.551Z        info    VictoriaMetrics/lib/promscrape/scraper.go:430   static_configs: added targets: 1, removed targets: 0; total targets: 1
2024-02-08T21:08:48.606Z        error   VictoriaMetrics/app/vmagent/remotewrite/client.go:465   unexpected status code received after sending a block with size 92156 bytes to "1:secret-url" during retry #2: 401; response body="\n"; re-sending the block in 4.109 seconds
2024-02-08T21:08:48.634Z        error   VictoriaMetrics/app/vmagent/remotewrite/client.go:465   unexpected status code received after sending a block with size 9853 bytes to "1:secret-url" during retry #2: 401; response body="\n"; re-sending the block in 4.180 seconds
2024-02-08T21:08:48.642Z        error   VictoriaMetrics/app/vmagent/remotewrite/client.go:465   unexpected status code received after sending a block with size 9854 bytes to "1:secret-url" during retry #2: 401; response body="\n"; re-sending the block in 4.178 seconds
2024-02-08T21:08:48.723Z        error   VictoriaMetrics/app/vmagent/remotewrite/client.go:465   unexpected status code received after sending a block with size 92156 bytes to "1:secret-url" during retry #2: 401; response body="\n"; re-sending the block in 4.370 seconds
2024-02-08T21:08:52.717Z        error   VictoriaMetrics/app/vmagent/remotewrite/client.go:465   unexpected status code received after sending a block with size 92156 bytes to "1:secret-url" during retry #3: 401; response body="\n"; re-sending the block in 8.217 seconds
2024-02-08T21:08:52.819Z        error   VictoriaMetrics/app/vmagent/remotewrite/client.go:465   unexpected status code received after sending a block with size 9853 bytes to "1:secret-url" during retry #3: 401; response body="\n"; re-sending the block in 8.361 seconds
2024-02-08T21:08:52.821Z        error   VictoriaMetrics/app/vmagent/remotewrite/client.go:465   unexpected status code received after sending a block with size 9854 bytes to "1:secret-url" during retry #3: 401; response body="\n"; re-sending the block in 8.357 seconds
2024-02-08T21:08:53.095Z        error   VictoriaMetrics/app/vmagent/remotewrite/client.go:465   unexpected status code received after sending a block with size 92156 bytes to "1:secret-url" during retry #3: 401; response body="\n"; re-sending the block in 8.740 seconds

Additional information

I know the VM server is working because if I use Grafana Agent to write to the VM server it works fine so it would appear to be some error with VMAgent, specifically that it does not seem to be authenticating properly since it's getting a 401 HTTP status code. If I remove the auth user/pass from VM and then from the agent the errors described in this ticket do not happen

OmegaZero avatar Feb 08 '24 21:02 OmegaZero

Hello, for remote write authentication you have to use a different flag for vmagent, instead of -httpAuth - remoteWrite.basicAuth:

vmagent-windows-amd64-prod.exe -promscrape.config=C:\Monitoring\vmagent\prometheus.yml -remoteWrite.basicAuth.usernamee=grafana -remoteWrite.basicAuth.password=oWsONTEalEtUdGeROXYglEmAKyhwwB -remoteWrite.url=http://localhost:8428/api/v1/write

f41gh7 avatar Feb 09 '24 10:02 f41gh7

I'm closing this one as the original question is answered. Feel free to re-open it if you think the comment here does not work.

jiekun avatar Jul 14 '24 05:07 jiekun