How to handle prometheus remote_write case with influxdb-relay

[Alexvianet]

how should i configure relay in prometheus case prometheus config:

# Remote write configuration (for Graphite, OpenTSDB, or InfluxDB).
remote_write:
  - url: "http://relay.Loadbalancer:9096/api/v1/prom/write?u=foo&p=bar&db=prometheus"

relay config:

bind-addr = "0.0.0.0:9096"
# Array of InfluxDB instances to use as backends for Relay.
output = [

 { name="influxdb1", location="http://10.35.96.144:8086/write", timeout="10s" },
 { name="influxdb2", location="http://10.35.96.145:8086/write", timeout="10s" },

]

Loadbalancer nginx config:

http {
  client_max_body_size 20M;
  sendfile    on;
  tcp_nopush  on;
  tcp_nodelay on;

  upstream influxdb-relay {

    server relay:9096 fail_timeout=0;
  }
  upstream influxdb {

    server influxdb1:8086 fail_timeout=0;
    server influxdb2:8086 fail_timeout=0;
  }

  server {
    listen                  9096;
    server_name             relay.Loadbalancer;

    location /write {
      limit_except POST {
        deny all;
                        }
      proxy_pass http://influxdb-relay;
                    }
    location /api/v1/prom/write {
      limit_except POST {
        deny all;
                        }
      proxy_pass http://influxdb-relay;
                    }
        }

  server {
    listen                  8086;
    server_name             influxdb.Loadbalancer;
    location /query {
      limit_except GET {
        deny all;
    }

    proxy_pass http://influxdb;

  }
    location /read {
      limit_except GET {
        deny all;
    }

    proxy_pass http://influxdb;

  }
    location /api/v1/prom/read {
      limit_except GET {
        deny all;
    }

    proxy_pass http://influxdb;

  }
 }
}

and get error 2018/06/01 14:08:02 [error] 10360#0: *69959 readv() failed (104: Connection reset by peer) while reading upstream, client: prometheus_host, server: relay.Loadbalancer, request: "POST /api/v1/prom/write?u=foo&p=bar&db=prometheus HTTP/1.1", upstream: "http://relay:9096/api/v1/prom/write?u=foo&p=bar&db=prometheus", host: "relay.Loadbalancer:9096"

[Alexvianet]

Run on relay server: curl -i -XPOST 'http://localhost:9096/api/v1/prom/write?u=foo&p=bar&db=prometheus' --data-binary 'cpu_load_short,host=server01,region=us-west value=22.64 1434055562000000000'

gives:

HTTP/1.1 404 Not Found
Content-Length: 35
Content-Type: application/json
Date: Mon, 04 Jun 2018 11:32:54 GMT

{"error":"invalid write endpoint"}

[angryp]

Hi @Alexvianet!

I have exactly same situation here and I assume Prometheus does not work with influxdb-relay at all. Assuming influxdb-relay has not been updated for 2 years now, I doubt it will work at all. I had also tried out remote_storage_adapter by Prometheus - it does create field names, but does not populate them with data. Series are empty; measurements exist, but they are empty. In short, influxdb-relay does not work with it either.

P.S. At least the curl error you mentioned in the second post can be easily fixed by commenting out relay/http.go:122 function section (just for tests of course), but it won't help writing the data.

[knope]

My team is also hitting a pain point on this. I recently ditched the remote_storage_adapter in favor of the native influx support, however the relay is a must to ensure the data captured is reliable.

• In my use case i have an ALB listening on :7076 and sorting reads directly to the influxdb, with writes going to the relay listener (as per the recommended model) relevant lines in relay.toml i.e.:

output = [
    { name="local-influxdb", location="http://10.224.16.147:8086/api/v1/prom/write", timeout="10s"  },
    { name="remote-influxdb", location="http://10.224.16.200:8086/api/v1/prom/write", timeout="10s"  }]

• Prometheus remotes are configured as such:

remote_write:
  - url: "<http://url>:7076/api/v1/prom/write?db=prometheus"
remote_read:
  - url: "<http://url>:7076/api/v1/prom/read?db=prometheus"

• Relay service is healthy, the health check returns healthy, reads are healthy against the influxdb, however the <http://url>:7076/api/v1/prom/write?db=prometheus data is lost, with:

err="server returned HTTP status 404 Not Found: {\"error\":\"invalid write endpoint\"}"

.. EDIT: found fork: https://github.com/influxdata/influxdb-relay/issues/67 testing for remote_write fixes ... .. EDIT2: errors still present with forked influxdb-relay, more in comments under 67^ .. EDIT3: for the time being, i've switched the writes to go through the deprecated storage_gateway via /write which the relay is happy with, and reads hit the api/v1/prom/read

[rockyluke]

Support for Prometheus remote write has been added on https://github.com/vente-privee/influxdb-relay (develop branch). Feel free to try it quickly!

[lucas-santoni]

@Alexvianet Did you have the chance to try it ? We continued to improve Prometheus support on our side and we will be glad to hear from you ! :D

[Alexvianet]

sorry no, we have moved away from influxdb to thanos with s3

On Fri, Nov 16, 2018, 18:47 Geographer [email protected] wrote:

@Alexvianet https://github.com/Alexvianet Did you have the chance to try it ? We continued to improve Prometheus support on our side and we will be glad to hear from you ! :D

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/influxdata/influxdb-relay/issues/66#issuecomment-439454232, or mute the thread https://github.com/notifications/unsubscribe-auth/AQrHRTD9WwZJIfFh4jPazGdkcTeKdeZ7ks5uvuwGgaJpZM4UWrOl .