influxdb-srelay help on build config to route on measurements

help on build config to route on measurements

Open afeshti opened this issue 1 year ago • 0 comments

Hello, thank you for this repo.

I am trying to create a srelay config that help me to route writes based on measurements on a influxdb-v1 input.

I can correcly duplicate to 2 influxdb in HA-Style this input:

net,host=vclient,interface=ens192,server=client err_in=0i,err_out=0i,drop_in=90i,drop_out=0i,bytes_sent=70922687456i,bytes_recv=161218632749i 1704468720000000000
database_acme_com,dbname=MTEMP_DB_DEV1,host=vclient,server=client,subsystem=writes,system=readandwrite value=39 1704468720000000000

with the following config:

[[influxdb]]
  name = "server1"
  location = "http://server1:8186/"
  timeout = "10s"

[[influxdb]]
  name = "server2"
  location = "http://server2:8086/"
  timeout = "10s"

[[influxcluster]]
  name  = "ha_cluster"
  members = ["server1","server2"]
  log-file = "/var/log/ha_cluster.log"
  log-level = "debug"
  type = "HA"
  query-router-endpoint-api = ["http://server1:4090/api/queryactive","http://server2:4090/api/queryactive"]

[[http]]
  name = "ha-relay"
  bind-addr = "0.0.0.0:8085"
  log-file = "/var/log/http_harelay_8085.log"
  log-level = "debug"
  access-log = "/var/log/ha_access_8085.log"

  rate-limit = 1000000
  burst-limit = 2000000

  [[http.endpoint]]
    uri=["/write"]
    source_format="ILP"
    type = "WR"

    [[http.endpoint.route]]
      name="any_write"
      log-file = "/var/log/write_metrics.log"
      level="http"

      [[http.endpoint.route.filter]]
        name="pass_all"
        key="db"
        match=".*"

      [[http.endpoint.route.rule]]
        name="route_all"
        action="route"
        key="db"
        match=".*"
        to_cluster="ha_cluster"

host-file:

10.224.90.177 server2
10.224.90.188 server1

network trace, from the relay to the remote hosts:

ngrep -t -qd any -W byline \( port 8186 or port 8086 \) and \( host 10.224.90.177 or host 10.224.90.188 \)
interface: any
filter: ( ( port 8186 or port 8086 ) and ( host 10.224.90.177 or host 10.224.90.188 ) ) and (ip || ip6)

T 2024/01/12 13:05:10.249199 10.224.90.177:36986 -> 10.224.90.177:8086 [AP] #11
POST /write?consistency=any&db=dmz HTTP/1.1.
Host: server2:8086.
User-Agent: influxdb-smart-relay.
Content-Length: 294.
Content-Type: text/plain.
Accept-Encoding: gzip.
.
net,host=vclient,interface=ens192,server=client err_in=0i,err_out=0i,drop_in=90i,drop_out=0i,bytes_sent=70922687456i,bytes_recv=161218632749i 1704468720000000000
database_acme_com,dbname=MTEMP_DB_DEV1,host=vclient,server=client,subsystem=writes,system=readandwrite value=39 1704468720000000000


T 2024/01/12 13:05:10.249241 10.224.90.177:33736 -> 10.224.90.188:8186 [AP] #13
POST /write?consistency=any&db=dmz HTTP/1.1.
Host: server1:8186.
User-Agent: influxdb-smart-relay.
Content-Length: 294.
Content-Type: text/plain.
Accept-Encoding: gzip.
.
net,host=vclient,interface=ens192,server=client err_in=0i,err_out=0i,drop_in=90i,drop_out=0i,bytes_sent=70922687456i,bytes_recv=161218632749i 1704468720000000000
database_acme_com,dbname=MTEMP_DB_DEV1,host=vclient,server=client,subsystem=writes,system=readandwrite value=39 1704468720000000000


T 2024/01/12 13:05:10.358572 10.224.90.188:8186 -> 10.224.90.177:33736 [AP] #15
HTTP/1.1 204 No Content.
Content-Type: application/json.
Request-Id: d578ea57-b142-11ee-8c50-801844f28aa8.
X-Influxdb-Build: OSS.
X-Influxdb-Version: 1.8.10.
X-Request-Id: d578ea57-b142-11ee-8c50-801844f28aa8.
Date: Fri, 12 Jan 2024 12:05:10 GMT.
.


T 2024/01/12 13:05:10.360447 10.224.90.177:8086 -> 10.224.90.177:36986 [AP] #17
HTTP/1.1 204 No Content.
Content-Type: application/json.
Request-Id: d578ecb8-b142-11ee-b781-00262d0017a8.
X-Influxdb-Build: OSS.
X-Influxdb-Version: 1.8.10.
X-Request-Id: d578ecb8-b142-11ee-b781-00262d0017a8.
Date: Fri, 12 Jan 2024 12:05:10 GMT.
.

But what I am trying to do is to rotate the input based on measurements. So, for ie, I would like to rotate:

net,host=vclient,interface=ens192,server=client err_in=0i,err_out=0i,drop_in=90i,drop_out=0i,bytes_sent=70922687456i,bytes_recv=161218632749i 1704468720000000000

to server1

and

database_acme_com,dbname=MTEMP_DB_DEV1,host=vclient,server=client,subsystem=writes,system=readandwrite value=39 1704468720000000000

to server2.

I have tried a simple config like that:

[[influxdb]]
  name = "server1"
  location = "http://server1:8186/"
  timeout = "10s"

[[influxdb]]
  name = "server2"
  location = "http://server2:8086/"
  timeout = "10s"

[[influxcluster]]
  name  = "server1"
  members = ["server1"]
  log-file = "/var/log/single_server1.log"
  log-level = "debug"
  type = "Single"

[[influxcluster]]
  name  = "server2"
  members = ["server2"]
  log-file = "/var/log/single_server2.log"
  log-level = "debug"
  type = "Single"

# HTTP Server
[[http]]
  name = "ha-relay"
  bind-addr = "0.0.0.0:8085"
  log-file = "/var/log/http_harelay_8085.log"
  log-level = "debug"
  access-log = "/var/log/ha_access_8085.log"

  rate-limit = 1000000
  burst-limit = 2000000

  [[http.endpoint]]
    uri=["/write"]
    source_format="ILP"
    type = "WR"

    [[http.endpoint.route]]
      name="any_write"
      log-file = "/var/log/write_metrics.log"
      level="data"

      [[http.endpoint.route.filter]]
        name="pass_all"
        key="db"
        match=".*"

      [[http.endpoint.route.rule]]
        name="database"
        action="route"
        key="measurement"
        match="database.*"
        to_cluster="server2"

      [[http.endpoint.route.rule]]
        name="os"
        action="route"
        key="measurement"
        match="(?:cpu.*|disk.*|mem.*|net.*|processes.*|swap.*|system.*)"
        to_cluster="server1"

but the route is not happening, and the logs says:

==> /var/log/http_harelay_8085.log <==
2024-01-12 13:14:34 DBG IN REQUEST:&{Method:POST URL:/write?consistency=any&db=dmz Proto:HTTP/1.1 ProtoMajor:1 ProtoMinor:1 Header:map[Accept:[*/*] Content-Type:[text/plain; charset=utf-8] User-Agent:[Telegraf/1.28.5 Go/1.21.4]] Body:0xc0001e8080 GetBody:<nil> ContentLength:-1 TransferEncoding:[chunked] Close:false Host:influxdb-sv02.acme.com:8085 Form:map[] PostForm:map[] MultipartForm:<nil> Trailer:map[] RemoteAddr:10.126.66.121:39440 RequestURI:/write?consistency=any&db=dmz TLS:<nil> Cancel:<nil> Response:<nil> ctx:0xc0001cc210}
2024-01-12 13:14:34 DBG ----------------------INIT rateMiddleware-----------------------
2024-01-12 13:14:34 DBG ----------------------INIT logMiddleWare------------------------
2024-01-12 13:14:34 DBG ----------------------INIT queryMiddleWare------------------------
2024-01-12 13:14:34 DBG ----------------------INIT bodyMiddleWare------------------------
2024-01-12 13:14:34 DBG Processing [0][WR] endpoint [/write]
2024-01-12 13:14:34 INF Init Processing Endpoint /write
2024-01-12 13:14:34 DBG Processing WRITE route 0 , any_write
2024-01-12 13:14:34 DBG Processing WRITE route 0 , &{cfg:0xc00012c980 DecodeFmt:ILP Type:WR log:0xc0001188a0 filters:[0xc0001783e0] rules:[0xc000153080 0xc0001530c0] DecodeData:0x809140}

==> /var/log/write_metrics.log <==
2024-01-12 13:14:34 INF No HTTP responses found on request route any_write

==> /var/log/http_harelay_8085.log <==
2024-01-12 13:14:34 DBG ----------------------END bodyMiddleWare------------------------
2024-01-12 13:14:34 DBG ----------------------END queryMiddleWare------------------------

==> /var/log/ha_access_8085.log <==
2024-01-12 13:14:34 INF  bk_duration_ms=9223372036854.775 duration_ms=1.604681 latency_ms=-9223372036854.775 method=POST referer= returnsize=0 source=10.126.66.121:39440 status=0 trace-route="http:ha-relay> rt:any_write> decode:ILP> rule:database> rule:os> " url=/write?consistency=any&db=dmz user=- user-agent="Telegraf/1.28.5 Go/1.21.4" write-points=2 write-size=294

==> /var/log/http_harelay_8085.log <==
2024-01-12 13:14:34 DBG ----------------------END logMiddleWare------------------------
2024-01-12 13:14:34 DBG ----------------------END rateMiddleware-----------------------

If I change the action from "route" to "route_db_from_data" it is almost ok:

[[influxdb]]
  name = "server1"
  location = "http://server1:8186/"
  timeout = "10s"

[[influxdb]]
  name = "server2"
  location = "http://server2:8086/"
  timeout = "10s"

[[influxcluster]]
  name  = "server1"
  members = ["server1"]
  log-file = "/var/log/single_server1.log"
  log-level = "debug"
  type = "Single"

[[influxcluster]]
  name  = "server2"
  members = ["server2"]
  log-file = "/var/log/single_server2.log"
  log-level = "debug"
  type = "Single"

# HTTP Server
[[http]]
  name = "ha-relay"
  bind-addr = "0.0.0.0:8085"
  log-file = "/var/log/http_harelay_8085.log"
  log-level = "debug"
  access-log = "/var/log/ha_access_8085.log"

  rate-limit = 1000000
  burst-limit = 2000000

  [[http.endpoint]]
    uri=["/write"]
    source_format="ILP"
    type = "WR"

    [[http.endpoint.route]]
      name="any_write"
      log-file = "/var/log/write_metrics.log"
      level="data"

      [[http.endpoint.route.filter]]
        name="pass_all"
        key="db"
        match=".*"

      [[http.endpoint.route.rule]]
        name="database"
        action="route_db_from_data"
        key="measurement"
        match="database.*"
        value="dmz"
        to_cluster="server2"

      [[http.endpoint.route.rule]]
        name="os"
        action="route_db_from_data"
        key="measurement"
        match="(?:cpu.*|disk.*|mem.*|net.*|processes.*|swap.*|system.*)"
        value="dmz"
        to_cluster="server1"

the route is happening, network trace:

ngrep -t -qd any -W byline \( port 8186 or port 8086 \) and \( host 10.224.90.177 or host 10.224.90.188 \)
interface: any
filter: ( ( port 8186 or port 8086 ) and ( host 10.224.90.177 or host 10.224.90.188 ) ) and (ip || ip6)

T 2024/01/12 13:18:30.896576 10.224.90.177:36996 -> 10.224.90.177:8086 [AP] #1
POST /write?chunked=&chunksize=&db=dmz&epoch=&p=&pretty=&q=&u= HTTP/1.1.
Host: server2:8086.
User-Agent: influxdb-smart-relay.
Content-Length: 132.
Content-Type: text/plain.
Accept-Encoding: gzip.
.
database_acme_com,dbname=MTEMP_DB_DEV1,host=vclient,server=client,subsystem=writes,system=readandwrite value=39 1704468720000000000


T 2024/01/12 13:18:31.010373 10.224.90.177:8086 -> 10.224.90.177:36996 [AP] #3
HTTP/1.1 204 No Content.
Content-Type: application/json.
Request-Id: b2b20480-b144-11ee-b7c6-00262d0017a8.
X-Influxdb-Build: OSS.
X-Influxdb-Version: 1.8.10.
X-Request-Id: b2b20480-b144-11ee-b7c6-00262d0017a8.
Date: Fri, 12 Jan 2024 12:18:31 GMT.
.


T 2024/01/12 13:18:31.010813 10.224.90.177:33746 -> 10.224.90.188:8186 [AP] #5
POST /write?chunked=&chunksize=&db=dmz&epoch=&p=&pretty=&q=&u= HTTP/1.1.
Host: server1:8186.
User-Agent: influxdb-smart-relay.
Content-Length: 162.
Content-Type: text/plain.
Accept-Encoding: gzip.
.
net,host=vclient,interface=ens192,server=client err_in=0i,err_out=0i,drop_in=90i,drop_out=0i,bytes_sent=70922687456i,bytes_recv=161218632749i 1704468720000000000


T 2024/01/12 13:18:31.122877 10.224.90.188:8186 -> 10.224.90.177:33746 [AP] #7
HTTP/1.1 204 No Content.
Content-Type: application/json.
Request-Id: b2c37ced-b144-11ee-8826-801844f28aa8.
X-Influxdb-Build: OSS.
X-Influxdb-Version: 1.8.10.
X-Request-Id: b2c37ced-b144-11ee-8826-801844f28aa8.
Date: Fri, 12 Jan 2024 12:18:31 GMT.
.

but the answer that the relay give to the client is HTTP 200 and not HTTP 204, that is the what the client expect:

curl -k -v 'http://influxdb-sv02.acme.it:8085/write?consistency=any&db=dmz' -H 'User-Agent: Telegraf/1.28.5 Go/1.21.4' -H 'Content-Type: text/plain; charset=utf-8' -H 'Transfer-Encoding: chunked' --data-binary @/tmp/post2
*   Trying 10.224.90.177:8085...
* Connected to influxdb-sv02.acme.it (10.224.90.177) port 8085 (#0)
> POST /write?consistency=any&db=dmz HTTP/1.1
> Host: influxdb-sv02.acme.it:8085
> Accept: */*
> User-Agent: Telegraf/1.28.5 Go/1.21.4
> Content-Type: text/plain; charset=utf-8
> Transfer-Encoding: chunked
>
* Mark bundle as not supporting multiuse
**< HTTP/1.1 200 OK**
< Date: Fri, 12 Jan 2024 12:18:31 GMT
< Content-Length: 0
<
* Connection #0 to host influxdb-sv02.acme.it left intact

That HTTP 204 is correctly answered when I use the first HA-Style srelay config.

Here the logs in debug in that latest case:

==> /var/log/http_harelay_8085.log <==
2024-01-12 13:21:12 DBG IN REQUEST:&{Method:POST URL:/write?consistency=any&db=dmz Proto:HTTP/1.1 ProtoMajor:1 ProtoMinor:1 Header:map[Accept:[*/*] Content-Type:[text/plain; charset=utf-8] User-Agent:[Telegraf/1.28.5 Go/1.21.4]] Body:0xc00023c5c0 GetBody:<nil> ContentLength:-1 TransferEncoding:[chunked] Close:false Host:influxdb-sv02.acme.it:8085 Form:map[] PostForm:map[] MultipartForm:<nil> Trailer:map[] RemoteAddr:10.126.66.121:41886 RequestURI:/write?consistency=any&db=dmz TLS:<nil> Cancel:<nil> Response:<nil> ctx:0xc000231380}
2024-01-12 13:21:12 DBG ----------------------INIT rateMiddleware-----------------------
2024-01-12 13:21:12 DBG ----------------------INIT logMiddleWare------------------------
2024-01-12 13:21:12 DBG ----------------------INIT queryMiddleWare------------------------
2024-01-12 13:21:12 DBG ----------------------INIT bodyMiddleWare------------------------
2024-01-12 13:21:12 DBG Processing [0][WR] endpoint [/write]
2024-01-12 13:21:12 INF Init Processing Endpoint /write
2024-01-12 13:21:12 DBG Processing WRITE route 0 , any_write
2024-01-12 13:21:12 DBG Processing WRITE route 0 , &{cfg:0xc000130b00 DecodeFmt:ILP Type:WR log:0xc0001388a0 filters:[0xc000192480] rules:[0xc00016f100 0xc00016f140] DecodeData:0x809140}

==> /var/log/write_metrics.log <==
2024-01-12 13:21:12 INF processing output for db dmz : # 1 Points
2024-01-12 13:21:12 INF Handle DB route Write to dmz.....
2024-01-12 13:21:12 INF processing output for db dmz : # 1 Points
2024-01-12 13:21:12 INF Handle DB route Write to dmz.....

Please, can you help me to build the right config ?

I am using the 0.6.0 version:

/root/go/bin/influxdb-srelay -version
influxdb-srelay version 0.6.0

Thanks Aldo

Jan 12 '24 12:01 afeshti

influxdb-srelay influxdb-srelay copied to clipboard

help on build config to route on measurements

influxdb-srelay
influxdb-srelay copied to clipboard