dnstap: Remote socket support

[danielloader]

A note for the community

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Use Cases

I'll start by saying thanks for getting this far! Vector has been a joy to work with thus far in lots of log shipping situations.

That being said I'm facing the requirement of shipping dnstap logs off the resolvers to another service, on kubernetes nodes via a service using TCP.

Listening to a unix domain socket isn't feasible in this scenario, as there's an imperative to offload logging to another host to reduce performance impact of powerdns logging, and would necessitate a TCP option.

Attempted Solutions

Well, at first I've attempted a sidecar solution using a TCP to UDS redirect:

socat TCP-LISTEN:1234,reuseaddr,fork UNIX-CLIENT:/var/dnstap.sock

This works, as expected, as long as you share an in memory mount between containers in the same pod.

It's just quite fragile in comparison to letting Vector handle the TCP > dnstap ingestion.

Proposal

Enable the dnstap source to be able to take TCP connections in addition to UDS connections.

References

https://github.com/vectordotdev/vector/issues/2516

Version

vector 0.25.1 (aarch64-unknown-linux-musl 9125a99 2022-11-07)

[james-stevens]

Seeing as both the syslog and socket sources have TCP & Unix socket support, it would seem to me a relatively trivial copy-paste operation to add this feature, and it's hard to see any justification to not have it. When containerising services, only having unix socket support is inconvenient.

Alternative solutions to using socat (a popular choice for solving this issue) would be using "socket activation" in systemd, an example config can be found here. Or run a local copy of vector for the unix socket, then use vector-to-vector pipelining to pass the dnstap data to a remote vector.

[peterand-pa]

Thumbs up

[peterand-pa]

I had an idea of using type: socket , but it's not decoding for some reason ... Config snippet: sources: dnstap: type: socket address: 0.0.0.0:6053 mode: tcp decoding: codec: protobuf protobuf: message_type: dnstap.Dnstap desc_file: /etc/vector/dnstap.desc host_key: host

[james-stevens]

I had an idea of using type: socket , but it's not decoding for some reason ... Config snippet: sources: dnstap: type: socket address: 0.0.0.0:6053 mode: tcp decoding: codec: protobuf protobuf: message_type: dnstap.Dnstap desc_file: /etc/vector/dnstap.desc host_key: host

I couldn't get the protobuf socket type to work - tried it with PowerDNS Recursor, which has a proprietary protobuf output as well as dnstap.

This is me moaning about that issue... https://github.com/vectordotdev/vector/discussions/18897

But for dnstap the dedicated dnstap source that works fine - so we just use that https://vector.dev/docs/reference/configuration/sources/dnstap/

You obv can get unix / tcp socket issues (as per this thread) - to solve that we run vector on the DNS server box, then stream vector to vector to get it off-box. Works fine.

[johnhtodd]

This is complete for DNSTAP - see https://github.com/vectordotdev/vector/issues/19735 - done and working well for a few hundred billion queries here already.

The second bug you note (PowerDNS format not working) you outlined in https://github.com/vectordotdev/vector/issues/19323 and maybe we'll take a swing at that as well.

[jszwedko]

👍 this is closed by https://github.com/vectordotdev/vector/issues/19735