DnsServer icon indicating copy to clipboard operation
DnsServer copied to clipboard
verified publisher icon TechnitiumSoftware

LogExporterApp JSON/HTTP output using jsonl

Open vanhecke opened this issue 6 months ago • 6 comments

The LogExporterApp offers event forwarding via HTTPS POST messages which is an excellent & modern way to forward logs to cloud logging platforms.

However the http client is creating sub-optimal payloads that are not natively supported by many of these systems.

Issues:

  1. Duplicate data (MessageTemplate and RenderedMessage seem identical) -> Suggest to remove
  2. Custom format / nesting offers no value -> Suggest to use jsonl

Current payload example:

[
  {
    "Timestamp": "2025-07-11T16:46:15.4306550Z",
    "Level": "Information",
    "MessageTemplate": "{\"timestamp\":\"2025-07-11T16:46:12.925Z\",\"clientIp\":\"10.13.37.24\",\"protocol\":\"Udp\",\"responseType\":\"Cached\",\"responseCode\":\"NoError\",\"question\":{\"questionName\":\"ocsp.godaddy.com\",\"questionType\":\"A\",\"questionClass\":\"IN\"},\"answers\":[{\"name\":\"ocsp.godaddy.com\",\"recordType\":\"CNAME\",\"recordClass\":\"IN\",\"recordTtl\":3596,\"recordData\":\"ocsp.godaddy.com.akadns.net.\",\"dnssecStatus\":\"Insecure\"},{\"name\":\"ocsp.godaddy.com.akadns.net\",\"recordType\":\"A\",\"recordClass\":\"IN\",\"recordTtl\":56,\"recordData\":\"192.124.249.22\",\"dnssecStatus\":\"Insecure\"},{\"name\":\"ocsp.godaddy.com.akadns.net\",\"recordType\":\"A\",\"recordClass\":\"IN\",\"recordTtl\":56,\"recordData\":\"192.124.249.41\",\"dnssecStatus\":\"Insecure\"},{\"name\":\"ocsp.godaddy.com.akadns.net\",\"recordType\":\"A\",\"recordClass\":\"IN\",\"recordTtl\":56,\"recordData\":\"192.124.249.23\",\"dnssecStatus\":\"Insecure\"},{\"name\":\"ocsp.godaddy.com.akadns.net\",\"recordType\":\"A\",\"recordClass\":\"IN\",\"recordTtl\":56,\"recordData\":\"192.124.249.24\",\"dnssecStatus\":\"Insecure\"},{\"name\":\"ocsp.godaddy.com.akadns.net\",\"recordType\":\"A\",\"recordClass\":\"IN\",\"recordTtl\":56,\"recordData\":\"192.124.249.36\",\"dnssecStatus\":\"Insecure\"}]}",
    "RenderedMessage": "{\"timestamp\":\"2025-07-11T16:46:12.925Z\",\"clientIp\":\"10.13.37.24\",\"protocol\":\"Udp\",\"responseType\":\"Cached\",\"responseCode\":\"NoError\",\"question\":{\"questionName\":\"ocsp.godaddy.com\",\"questionType\":\"A\",\"questionClass\":\"IN\"},\"answers\":[{\"name\":\"ocsp.godaddy.com\",\"recordType\":\"CNAME\",\"recordClass\":\"IN\",\"recordTtl\":3596,\"recordData\":\"ocsp.godaddy.com.akadns.net.\",\"dnssecStatus\":\"Insecure\"},{\"name\":\"ocsp.godaddy.com.akadns.net\",\"recordType\":\"A\",\"recordClass\":\"IN\",\"recordTtl\":56,\"recordData\":\"192.124.249.22\",\"dnssecStatus\":\"Insecure\"},{\"name\":\"ocsp.godaddy.com.akadns.net\",\"recordType\":\"A\",\"recordClass\":\"IN\",\"recordTtl\":56,\"recordData\":\"192.124.249.41\",\"dnssecStatus\":\"Insecure\"},{\"name\":\"ocsp.godaddy.com.akadns.net\",\"recordType\":\"A\",\"recordClass\":\"IN\",\"recordTtl\":56,\"recordData\":\"192.124.249.23\",\"dnssecStatus\":\"Insecure\"},{\"name\":\"ocsp.godaddy.com.akadns.net\",\"recordType\":\"A\",\"recordClass\":\"IN\",\"recordTtl\":56,\"recordData\":\"192.124.249.24\",\"dnssecStatus\":\"Insecure\"},{\"name\":\"ocsp.godaddy.com.akadns.net\",\"recordType\":\"A\",\"recordClass\":\"IN\",\"recordTtl\":56,\"recordData\":\"192.124.249.36\",\"dnssecStatus\":\"Insecure\"}]}"
  },
  {
    "Timestamp": "2025-07-11T16:46:15.4309626Z",
    "Level": "Information",
    "MessageTemplate": "{\"timestamp\":\"2025-07-11T16:46:13.284Z\",\"clientIp\":\"10.13.37.24\",\"protocol\":\"Udp\",\"responseType\":\"Cached\",\"responseCode\":\"NoError\",\"question\":{\"questionName\":\"storage.googleapis.com\",\"questionType\":\"AAAA\",\"questionClass\":\"IN\"},\"answers\":[{\"name\":\"storage.googleapis.com\",\"recordType\":\"AAAA\",\"recordClass\":\"IN\",\"recordTtl\":222,\"recordData\":\"2a00:1450:400c:c02::cf\",\"dnssecStatus\":\"Insecure\"},{\"name\":\"storage.googleapis.com\",\"recordType\":\"AAAA\",\"recordClass\":\"IN\",\"recordTtl\":222,\"recordData\":\"2a00:1450:400c:c09::cf\",\"dnssecStatus\":\"Insecure\"},{\"name\":\"storage.googleapis.com\",\"recordType\":\"AAAA\",\"recordClass\":\"IN\",\"recordTtl\":222,\"recordData\":\"2a00:1450:400c:c0c::cf\",\"dnssecStatus\":\"Insecure\"},{\"name\":\"storage.googleapis.com\",\"recordType\":\"AAAA\",\"recordClass\":\"IN\",\"recordTtl\":222,\"recordData\":\"2a00:1450:400c:c07::cf\",\"dnssecStatus\":\"Insecure\"}]}",
    "RenderedMessage": "{\"timestamp\":\"2025-07-11T16:46:13.284Z\",\"clientIp\":\"10.13.37.24\",\"protocol\":\"Udp\",\"responseType\":\"Cached\",\"responseCode\":\"NoError\",\"question\":{\"questionName\":\"storage.googleapis.com\",\"questionType\":\"AAAA\",\"questionClass\":\"IN\"},\"answers\":[{\"name\":\"storage.googleapis.com\",\"recordType\":\"AAAA\",\"recordClass\":\"IN\",\"recordTtl\":222,\"recordData\":\"2a00:1450:400c:c02::cf\",\"dnssecStatus\":\"Insecure\"},{\"name\":\"storage.googleapis.com\",\"recordType\":\"AAAA\",\"recordClass\":\"IN\",\"recordTtl\":222,\"recordData\":\"2a00:1450:400c:c09::cf\",\"dnssecStatus\":\"Insecure\"},{\"name\":\"storage.googleapis.com\",\"recordType\":\"AAAA\",\"recordClass\":\"IN\",\"recordTtl\":222,\"recordData\":\"2a00:1450:400c:c0c::cf\",\"dnssecStatus\":\"Insecure\"},{\"name\":\"storage.googleapis.com\",\"recordType\":\"AAAA\",\"recordClass\":\"IN\",\"recordTtl\":222,\"recordData\":\"2a00:1450:400c:c07::cf\",\"dnssecStatus\":\"Insecure\"}]}"
  }
]

The preferred layout would be to use the jsonl format (info: jsontools.com/what-is-jsonl and jsonlines.org). And overall reduce the output.

Suggestion (using jsonl format, and only including "RenderedMessage"):

{"timestamp":"2025-07-11T16:46:12.925Z","clientIp":"10.13.37.24","protocol":"Udp","responseType":"Cached","responseCode":"NoError","question":{"questionName":"ocsp.godaddy.com","questionType":"A","questionClass":"IN"},"answers":[{"name":"ocsp.godaddy.com","recordType":"CNAME","recordClass":"IN","recordTtl":3596,"recordData":"ocsp.godaddy.com.akadns.net.","dnssecStatus":"Insecure"},{"name":"ocsp.godaddy.com.akadns.net","recordType":"A","recordClass":"IN","recordTtl":56,"recordData":"192.124.249.22","dnssecStatus":"Insecure"},{"name":"ocsp.godaddy.com.akadns.net","recordType":"A","recordClass":"IN","recordTtl":56,"recordData":"192.124.249.41","dnssecStatus":"Insecure"},{"name":"ocsp.godaddy.com.akadns.net","recordType":"A","recordClass":"IN","recordTtl":56,"recordData":"192.124.249.23","dnssecStatus":"Insecure"},{"name":"ocsp.godaddy.com.akadns.net","recordType":"A","recordClass":"IN","recordTtl":56,"recordData":"192.124.249.24","dnssecStatus":"Insecure"},{"name":"ocsp.godaddy.com.akadns.net","recordType":"A","recordClass":"IN","recordTtl":56,"recordData":"192.124.249.36","dnssecStatus":"Insecure"}]}
{"timestamp":"2025-07-11T16:46:13.284Z","clientIp":"10.13.37.24","protocol":"Udp","responseType":"Cached","responseCode":"NoError","question":{"questionName":"storage.googleapis.com","questionType":"AAAA","questionClass":"IN"},"answers":[{"name":"storage.googleapis.com","recordType":"AAAA","recordClass":"IN","recordTtl":222,"recordData":"2a00:1450:400c:c02::cf","dnssecStatus":"Insecure"},{"name":"storage.googleapis.com","recordType":"AAAA","recordClass":"IN","recordTtl":222,"recordData":"2a00:1450:400c:c09::cf","dnssecStatus":"Insecure"},{"name":"storage.googleapis.com","recordType":"AAAA","recordClass":"IN","recordTtl":222,"recordData":"2a00:1450:400c:c0c::cf","dnssecStatus":"Insecure"},{"name":"storage.googleapis.com","recordType":"AAAA","recordClass":"IN","recordTtl":222,"recordData":"2a00:1450:400c:c07::cf","dnssecStatus":"Insecure"}]}

I wanted to propose this to the contributors before attempting to make the code change myself. Thanks

vanhecke avatar Jul 11 '25 17:07 vanhecke

Thanks for the post. This app was implemented by @zbalkan and I do not have this app in production in any of my setup so not sure how this JSON output is being rendered. There is no such explicit code in the app which generates this JSON output. It seems that the Serilog library which is being used for this is generating this final JSON output. @zbalkan probably has this running in production so he may be in a better position to explain this.

Regarding changing the format, its really not feasible now that the app is being used in production. Also JSONL format seems to be about storing multiple entries per line in a file, however these logs are generated for each DNS request and are sent individually to the server immediately or within few seconds.

ShreyasZare avatar Jul 12 '25 08:07 ShreyasZare

Thanks for your feedback.

Also JSONL format seems to be about storing multiple entries per line in a file, however these logs are generated for each DNS request and are sent individually to the server immediately or within few seconds.

The app reads up to 1000 logs from a queue, pushes them to the destination, and then sleeps 10 seconds. https://github.com/TechnitiumSoftware/DnsServer/blob/3a7636ac3b1c221521f4031bb185277bbc85f068/Apps/LogExporterApp/App.cs#L46-L47

So it's roughly every 10 seconds. And up to 1000 logs.

vanhecke avatar Jul 12 '25 09:07 vanhecke

Also JSONL format seems to be about storing multiple entries per line in a file, however these logs are generated for each DNS request and are sent individually to the server immediately or within few seconds.

The app reads up to 1000 logs from a queue, pushes them to the destination, and then sleeps 10 seconds.

DnsServer/Apps/LogExporterApp/App.cs

Lines 46 to 47 in 3a7636a const int QUEUE_TIMER_INTERVAL = 10000; const int BULK_INSERT_COUNT = 1000;

So it's roughly every 10 seconds. And up to 1000 logs.

Yes, the timer runs every 10 sec but if there are more than 1000 logs available, it will keep on sending them in batch of max 1000 logs. It "sleeps" (resets the timer to 10 sec) only when no logs are available. So, it does batching but the uploading task is done with Serilog here which does it individually. Not sure if it can be configured to do it differently.

ShreyasZare avatar Jul 12 '25 09:07 ShreyasZare

Hi @vanhecke,

May I ask what problem are you trying to solve?

zbalkan avatar Jul 12 '25 11:07 zbalkan

Hey @zbalkan I want to send the logs to a logging platform for security-related analytics that natively supports jsonl.

Sending them in jsonl would require less CPU cycles on both sides.

vanhecke avatar Jul 15 '25 21:07 vanhecke

So, I guess you already tested using the HTTP target and found that is the bottleneck, right? Can you please let us know about your test setup so that we can reproduce the issue?

Then, we can benchmark if the suggested change solved the issue causing the bottleneck.

zbalkan avatar Jul 15 '25 21:07 zbalkan