pdns-protobuf-receiver
pdns-protobuf-receiver copied to clipboard
PowerDNS
Feature/support all pb fields [WIP]
Hello,
This PR is a major rework on the Protobuf Parsing. The Published Json is quite changed.
It supports:
- Almost all fields (originalRequestorSubnet is missing / No use & test case on my side)
- Decodes of RRs & rdata
- Split functions for Common/Query/Response/RRs data
My entended use is to provide required info for our SIEM with RPZ support on the DNS side (PDNS-Recursor)
The Published Json is quite changed.
How compatible is it to the old format, for consumers?
Almost all fields (originalRequestorSubnet is missing / No use & test case on my side)
This field is important to many users, can you add it anyway? :)
The Published Json is quite changed.
How compatible is it to the old format, for consumers?
I'll try to raise a diff or something. BRB on the subject.
Almost all fields (originalRequestorSubnet is missing / No use & test case on my side)
This field is important to many users, can you add it anyway? :)
PR updated with the support of originalRequestorSubnet. It is based on Protobuf.py implementation in PDNS source tree.
JSON Produced by this PR
The main difference I see from current layout:
- Query related data have been moved to query dict
return_codehas moved to response dict
{
"dns_message": "CLIENT_RESPONSE",
"message_id": "f344c6f134c14351b87588d3e63bc8bf",
"socket_family": "IPv4",
"socket_protocol": "UDP",
"from_address": "10.0.0.1",
"to_address": "10.0.80.2",
"bytes": 247,
"dns_id": 34516,
"from_port": 54462,
"to_port": 53,
"response": {
"return_code": "NOERROR",
"rrs": [
{
"name": "a767.dscg3.akamai.net.",
"type": "A",
"class": "IN",
"ttl": 18,
"rdata": {
"address": "92.122.122.152"
},
"udr": false
},
{
"name": "a767.dscg3.akamai.net.",
"type": "A",
"class": "IN",
"ttl": 18,
"rdata": {
"address": "92.122.122.147"
},
"udr": false
}
]
},
"query_time": "2021-01-05T15:32:16.574470+00:00",
"response_time": "2021-01-05T15:32:16.575857+00:00",
"latency": 0.001387,
"query": {
"name": "ctldl.windowsupdate.com.",
"type": "A",
"class": "IN"
}
}
Reference JSON / Current one
{
"dns_message": "AUTH_QUERY",
"socket_family": "IPv6",
"socket protocol": "UDP",
"from_address": "0.0.0.0",
"to_address": "184.26.161.130",
"query_time": "2020-05-29 13:46:23.322",
"response_time": "1970-01-01 01:00:00.000",
"latency": 0,
"query_type": "A",
"query_name": "a13-130.akagtm.org.",
"return_code": "NOERROR",
"bytes": 4
}
Hello, this version works a lot better. The previous had plenty of error while parsing rdata (Sorry my dev env required some firewall opening at my company to be operational).
It has been tested so far on:
- A
- AAAA
- CNAME
- TXT
- MX
- NS
- PTR
- SRV
Not done: 'SPF': Almost impossible to test (Deprecated) cf: https://tools.ietf.org/html/rfc7208#page-11
@Habbie @dmachard Any thought or remarks ?
I have this version run for the past days and it works fine on my side with ~400 clients so far.
FWIW I've a similar requirement and have implemented this code. I'm pushing to the receiver from DNSDist and had to remove the lines:
"appliedPolicyTrigger": "applied_policy_trigger", # 8
"appliedPolicyHit": "applied_policy_hit", # 9
As I was getting:
May 05 15:28:06 pdnsr_host pdns_protobuf_receiver[2852]: 2021-05-05 15:28:06,897 Task exception was never retrieved
May 05 15:28:06 pdnsr_host pdns_protobuf_receiver[2852]: future: <Task finished coro=<cb_onpayload() done, defined at /usr/local/lib/python3.6/dist-packages/pdns_protobuf_receiver/receiver.py:278> exception=ValueError('Protocol message DNSResponse has no field appliedPolicyHit.',)>
May 05 15:28:06 pdnsr_host pdns_protobuf_receiver[2852]: Traceback (most recent call last):
May 05 15:28:06 pdnsr_host pdns_protobuf_receiver[2852]: File "/usr/local/lib/python3.6/dist-packages/pdns_protobuf_receiver/receiver.py", line 308, in cb_onpayload
May 05 15:28:06 pdnsr_host pdns_protobuf_receiver[2852]: parse_pb_msg_response(dns_pb2, dns_msg)
May 05 15:28:06 pdnsr_host pdns_protobuf_receiver[2852]: File "/usr/local/lib/python3.6/dist-packages/pdns_protobuf_receiver/receiver.py", line 203, in parse_pb_msg_response
May 05 15:28:06 pdnsr_host pdns_protobuf_receiver[2852]: elif dns_pb2.response.HasField(key):
May 05 15:28:06 pdnsr_host pdns_protobuf_receiver[2852]: ValueError: Protocol message DNSResponse has no field appliedPolicyHit.
and
May 05 15:26:32 pdnsr_host pdns_protobuf_receiver[1785]: 2021-05-05 15:26:32,257 Task exception was never retrieved
May 05 15:26:32 pdnsr_host pdns_protobuf_receiver[1785]: future: <Task finished coro=<cb_onpayload() done, defined at /usr/local/lib/python3.6/dist-packages/pdns_protobuf_receiver/receiver.py:279> exception=ValueError('Protocol message DNSResponse has no field appliedPolicyTrigger.',)>
May 05 15:26:32 pdnsr_host pdns_protobuf_receiver[1785]: Traceback (most recent call last):
May 05 15:26:32 pdnsr_host pdns_protobuf_receiver[1785]: File "/usr/local/lib/python3.6/dist-packages/pdns_protobuf_receiver/receiver.py", line 309, in cb_onpayload
May 05 15:26:32 pdnsr_host pdns_protobuf_receiver[1785]: parse_pb_msg_rrs(dns_pb2, dns_msg)
May 05 15:26:32 pdnsr_host pdns_protobuf_receiver[1785]: File "/usr/local/lib/python3.6/dist-packages/pdns_protobuf_receiver/receiver.py", line 204, in parse_pb_msg_response
May 05 15:26:32 pdnsr_host pdns_protobuf_receiver[1785]: if isinstance(val, str):
May 05 15:26:32 pdnsr_host pdns_protobuf_receiver[1785]: ValueError: Protocol message DNSResponse has no field appliedPolicyTrigger.
Other than that though it seems to be a step forward definitely. I'm seeing response.rrs.rdata.address fields as expected which is definitely useful for us.
@lwhitworth Hello I have been late on this. Looks like the PDNS Recursor PB messages are not in line with current dnsdist (1.5.1 at least) release. The last commit should fix the issue. Could you give it a try ?