DnsServer icon indicating copy to clipboard operation
DnsServer copied to clipboard

New major version of Log Exporter app

Open zbalkan opened this issue 1 month ago • 13 comments

  • Fixed typos in configuration file and the class.
  • Performance optimisations in concurrent data structures.
  • Added console/stdout export for comtainerized workloads or debugging.
  • Added TLD, BaseDomain and Subdomain fields for better analysis.
  • Added README.
  • Incremented minor version.

zbalkan avatar Nov 26 '25 11:11 zbalkan

Thanks for the PR. Will get it merged soon.

ShreyasZare avatar Nov 26 '25 13:11 ShreyasZare

There will be a major version soon, probably in two months. Until than this is just fixes.

zbalkan avatar Nov 26 '25 21:11 zbalkan

I decided to make some minor changes. Let's keep this as a draft.

zbalkan avatar Nov 27 '25 22:11 zbalkan

Now it is ready for review.

zbalkan avatar Nov 28 '25 22:11 zbalkan

I used a new library that has TLD database and can dissect FQDN into TLD, Registerable Domain, and subdomains. If Technitium DNS has this capability, and I missed that, I can replace it ASAP.

zbalkan avatar Nov 30 '25 07:11 zbalkan

I used a new library that has TLD database and can dissect FQDN into TLD, Registerable Domain, and subdomains. If Technitium DNS has this capability, and I missed that, I can replace it ASAP.

There is no built-in function for this available. This library would need to be updated frequently too since the TLD list changes frequently with new registrations and deletions.

ShreyasZare avatar Nov 30 '25 12:11 ShreyasZare

Okay. I finished my performance optimizations. It is now using less memory allocations and is faster. I am keeping that new Library for better metadata. I'll update LogExporterApp with minor versions if I have to update that dependency in the future.

zbalkan avatar Nov 30 '25 14:11 zbalkan

I will do some minor changes. I need a cache for the Nager.PublicSuffix results. It's slowing the resolution.

zbalkan avatar Dec 02 '25 20:12 zbalkan

New version now adds a new metadata for anyone interested in enrichment or analysis context: DomainInfo. It includes tld, baseDomain, and subdomain fields for enriched log processing.

{"answers":[{"dnssecStatus":"Disabled","name":"google.co.uk","recordClass":"IN","recordData":"216.58.210.131","recordTtl":142,"recordType":"A"}],"clientIp":"127.0.0.1","edns":[],"nameServer":"127.0.0.1","protocol":"Udp","question":{"questionClass":"IN","questionName":"google.co.uk","questionType":"A"},"responseCode":"NoError","responseRtt":14.2574,"responseType":"Recursive","timestamp":"2025-12-05T13:18:50.457Z","domainInfo":{"tld":"co.uk","baseDomain":"google.co.uk","subdomain":""}}

Here's the prettier version:

{
  "answers": [
    {
      "dnssecStatus": "Disabled",
      "name": "google.co.uk",
      "recordClass": "IN",
      "recordData": "216.58.210.131",
      "recordTtl": 142,
      "recordType": "A"
    }
  ],
  "clientIp": "127.0.0.1",
  "edns": [],
  "nameServer": "127.0.0.1",
  "protocol": "Udp",
  "question": {
    "questionClass": "IN",
    "questionName": "google.co.uk",
    "questionType": "A"
  },
  "responseCode": "NoError",
  "responseRtt": 14.2574,
  "responseType": "Recursive",
  "timestamp": "2025-12-05T13:18:50.457Z",
  "domainInfo": {
    "tld": "co.uk",
    "baseDomain": "google.co.uk",
    "subdomain": ""
  }
}

In order to improve performance of this additional processing task, I added a singleton called DomainCache, which is a SIEVE-algorithm-based cache implementation using Nager.PublicSuffix DomainInfo class.

zbalkan avatar Dec 05 '25 13:12 zbalkan

I will fix the copilot review issues, then make an enable/disable flag in config.

zbalkan avatar Dec 07 '25 07:12 zbalkan

Converted log exporter to a proper log forwarder with a pipeline. It has now a single enrichment tool but it can be easily extended with current capability. An approach similar toa plugin-to-a-plugin. Code is readable. It may not fit your style and formatting. I suggest providing a detailed .editorconfig file for consistency across contributors.

zbalkan avatar Dec 08 '25 20:12 zbalkan

BTW, I use SIEVE caching algorithm for enrichment addition here. If you think it is useful, I can make it a Generic SieveCache<T> so that it will be usable by other parts as well. I got the idea from. This article on BIND: https://www.isc.org/blogs/2025-sieve/

zbalkan avatar Dec 08 '25 21:12 zbalkan

BTW, I use SIEVE caching algorithm for enrichment addition here. If you think it is useful, I can make it a Generic SieveCache<T> so that it will be usable by other parts as well. I got the idea from. This article on BIND: https://www.isc.org/blogs/2025-sieve/

I am not sure where else it can be used so this can be done when required.

ShreyasZare avatar Dec 09 '25 06:12 ShreyasZare