Feature request: DNSTAP from TCP network socket

[johnhtodd]

A note for the community

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Use Cases

Currently, DNSTAP is ingested only via a UNIX socket. We have clusters of systems using network-delivered TCP streams of DNSTAP data. We do not do processing on each DNS resolver, as telemetry processing is intentionally distant from processing of DNS data to prevent side-effect issues. Therefore, we need some way to get the data from the separate DNS processing systems into the location-specific telemetry processor (Vector) via network transport, with as few configuration or sub-processing elements as possible. Having Vector ingest TCP network socket data seems to be a step towards that simplification.

Attempted Solutions

It is of course possible to use socat or other TCP-to-socket redirection strategies. However, this is less elegant and requires another component in the chain that adds failure points and possible performance bottlenecks which would be difficult to diagnose. It seems reasonable that Vector would ingest TCP data directly, as that is a primary delivery method described in DNSTAP specifications.

There is even a strong argument to support TLS unwrapping of DNSTAP messages as an option, as several of the most common platforms now support that encryption model (see https://dmachard.github.io/posts/0001-dnstap-testing/ ) and this is useful in cloud environments where untrusted network segments exist between the resolvers and the collection infrastructure.

Proposal

It would be useful to have Vector take in DNSTAP data directly via TCP socket, with some basic IP-based permit/deny security for IPv4 and IPv6 CIDR-based origins.

Optionally, a set of TLS behaviors (cert data, etc.) would be useful to support encrypted TCP data but that is optional. See https://dmachard.github.io/posts/0001-dnstap-testing/ for a list of systems and how they support TLS.

References

No response

Version

vector 0.34.2 (x86_64-unknown-linux-gnu d685a16 2024-01-02 14:59:54.890517436)

[peterand-pa]

Thumbs up

[jszwedko]

Thumbs up

Please use the emoji reaction on the issue itself :) That makes it easier to see +1s at a glance vs. comments.

[johnhtodd]

Notable comment: It is required that this would allow Vector to ingest data from many simultaneous DNSTAP streams at once from different sources. This seems like an obvious statement, but the current ingestion model via a socket may not actually support multiple origins in the same source declaration (as my experiments with socat in "fork" mode seem to be discovering.)

[esensar]

I have picked this ticket up, I think I am making some good progress on it, but I had a question about configuration. The way I see it (at least with the way I have implemented it), it has to be a breaking change. I made it similar to how socket source has mode field, meaning that mode field becomes required, making all current configurations using unix socket break, since they would need the mode field added to them.

Is that okay or should I aim for something that makes unix socket the default?

[jszwedko]

I have picked this ticket up, I think I am making some good progress on it, but I had a question about configuration. The way I see it (at least with the way I have implemented it), it has to be a breaking change. I made it similar to how socket source has mode field, meaning that mode field becomes required, making all current configurations using unix socket break, since they would need the mode field added to them.

Is that okay or should I aim for something that makes unix socket the default?

Great, thanks @esensar ! If it is possible to have unix be the default, that would be preferable to avoid the breaking change. If it turns out to not be possible, or very difficult, we can consider making the breaking change.