Enhancement: parse more DNSTAP message types

[johnhtodd]

A note for the community

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Use Cases

DNSTAP parsing was done some years ago, and I expect some use cases were not applied to it in testing. We get a lot of "WARNING" messages due to types of messages received which DNSTAP hasn't been configured to parse, and which are not finding their way correctly into our message streams. These are hopefully easy to fix by parsing more message types that are common in our visibility of the networks so they do not cause errors. Some of these errors appear to be due to unexpectedly blank fields, or values that are out of range for the existing DNSTAP code that probably need to be updated since the code was written.

Attempted Solutions

There are no solutions to this - DNSTAP parses and fails on the message types. The only solution is to have DNSTAP parse the messages correctly so as not to produce warnings. This will probably be an iterative process as we roll out Vector across a large number of locations that will see inventive new and unexpected DNS data formats and messages.

Proposal

I will attach to this ticket a file that contains the output of just the warning fields, and the raw data that created those warnings. I created the unparseable data by capturing all input messages out of my DNSTAP collector that had .rawData != null. Once I had the unparseable file, I needed to strip data out for privacy reasons, and created a summary file. The summary file was created like this:

cat /tmp/unparseable-2024-02-11.log |jq -r '.error,.responseData.rawData' > dnsstap-parse-errors

It should be possible to decode the raw query messages with this python script to see what the queries were originally, which then caused the replies causing the WARNING messages (note: over time, this list will produce fewer and fewer useful responses as the DNS changes)

#!/usr/bin/python3
import sys
import base64
import dns
import dns.message

line = sys.argv[1]
data = base64.b64decode(line)
m = dns.message.from_wire(data)
print(m)

References

No response

Version

vector 0.36.0 (x86_64-unknown-linux-gnu de24167 2024-02-09 04:01:38.285251865)

[johnhtodd]

Parse error summary file as described. dnstap-parse-errors.gz