go-dnscollector
go-dnscollector copied to clipboard
dmachard
Add arbitrary labels to data in collector and transformation stanzas
Is your feature request related to a problem? Please describe. It should be possible in any collector or transformation to set an arbitrary tag for use later in the pipeline to describe what has happened to data.
Describe the solution you'd like For all collectors and transformations, a simple declaration of a tag name and value would be ideal. For the purposes of this feature request, I'd call them "arbitrary tags" or "atags" for short. Something like:
{ "atags": { "specialstringnameimadeup": "Frobozz", "asamplevalue": 24 } }
Tags could be numeric or strings. Any number of key/value sets are supported. (maximum length?) If there is something already in the "atags" field with that exact name in the message before it was collected by this instance, the new value should overwrite it.
Describe alternatives you've considered Port ID or IP address is a weak way of identifying what has happened to a packet; I hate keeping those matrix mappings somewhere - it makes life exceptionally difficult.
Additional context The methods applied to any data packet should be able to be identified by something upstream from that source. Tagging seems like the most natural way to do it. I would like to know further down the chain what has happened to this data, or where it came from, and adding arbitrary tags seems like the way to do this.
Hi John, As always, your feature requests are really valuable to enhance this tool. I will review how to implement properly your request and come back to you with updates. Thank you for your contributions Denis
Looks like you did this in the pipeline branch, so ... excellent! It does appear to work, and I've done some preliminary testing with simple tag names.
A specific pull request will be come with tests and cleanup.
I realize that I haven't implemented exactly what is asked there. In the pipeline branch, I implemented a list instead of a dict.
{ "atags": [ "specialstringnameimadeup: Frobozz", "asamplevalue:24" ] }
instead of
{
"atags": {
"specialstringnameimadeup": "Frobozz",
"asamplevalue": 24 }
}
If the ordering is not important I can switch to dict as requested.
That's fine as you implemented it, and is actually more flexible as any string can be added instead of a specific model of key:value. I like the current method; no change needed!
But....
The reason I constructed it as a key:value pair was because I was envisioning future use as a filter method on these arbitrary structures, and having a fixed structure of key:value means that it would be easier to write filters if there is an understood structure. Using a ":" character to delineate the key and value makes things more easily parsed.
For instance, using the example above:
- name: tag-queries
dnsmessage:
matching:
include:
dnstap.operation: "CLIENT_RESPONSE"
atags.specialstringnameimadeup: "Frobozz"
policy: "drop-unmatched"
routes: [ go-to-frobozz-logger-step ]
Do you see another way around this that is just as flexible? Can matching work using key:values when the tag is structured in that form, and matching just isn't possible if the string structure isn't in key:value format?
The matching of any to key:value label has been added to the pipeline branch (in a generic way) It is not yet production-ready, but it should work.
If the key is a list, you need to provide the index or use '*' to search on each item in the list."
atags.tags.*: google
- name: match-tag
dnsmessage:
matching:
include:
atags.tags.*: google
drop-policy: "unmatched"
routes: [ outputfile, console ]
Amazing - will try out tomorrow! Does it support regular expressions?
yes and also match-source