data-prepper
data-prepper copied to clipboard
[BUG] aggregate put_all not working with nested keys
Describe the bug It seems that the put_all aggregate function only works with "root" keys, and do not add new nested keys.
For example, I have these 2 postfix events that I need to aggregate, based on /host/id and /email/local_id :
{
"@timestamp": "2024-01-19T13:25:35.503436Z",
"postfix": {
"nrcpt": "1",
"size": "17603"
},
"application": "postfix",
"host": {
"id": "95e0888c1df24d79a4bc827f9636cca7"
},
"email": {
"from": {
"address": "EMAIL_FROM"
},
"local_id": "7A20930009B"
},
"message": "queue active"
}
{
"@timestamp": "2024-01-19T13:25:35.527242Z",
"postfix": {
"relay": "localhost[::1]:24",
"delays": "0/0/0/0.02",
"dsn": "2.1.5",
"delay": "0.03",
"status": "sent"
},
"application": "postfix",
"host": {
"id": "95e0888c1df24d79a4bc827f9636cca7"
},
"email": {
"local_id": "7A20930009B",
"to": {
"address": "EMAIL_TO"
}
},
"test": "OK"
}
The put_all function only add keys that are at the '/' level of the JSON, and drop the other keys :
{
"@timestamp": "2024-01-19T13:25:35.503436Z",
"postfix": {
"nrcpt": "1",
"size": "17603"
},
"application": "postfix",
"host": {
"id": "95e0888c1df24d79a4bc827f9636cca7"
},
"email": {
"from": {
"address": "EMAIL_FROM"
},
"local_id": "7A20930009B"
},
"message": "queue active",
"test": "ok"
}
To Reproduce Steps to reproduce the behavior:
pipeline:
processor:
- aggregate:
identification_keys:
- "/host/id"
- "/email/local_id"
action:
put_all:
group_duration: "10s"
aggregate_when: '/email/local_id != null'
Expected behavior It needs to create the missing entries in the nested keys :
{
"@timestamp": "2024-01-19T13:25:35.503436Z",
"postfix": {
"nrcpt": "1",
"size": "17603",
"relay": "localhost[::1]:24",
"delays": "0/0/0/0.02",
"dsn": "2.1.5",
"delay": "0.03",
"status": "sent"
},
"application": "postfix",
"host": {
"id": "95e0888c1df24d79a4bc827f9636cca7"
},
"email": {
"from": {
"address": "EMAIL_FROM"
},
"to": {
"address": "EMAIL_TO"
}
"local_id": "7A20930009B"
},
"message": "queue active",
"test": "ok"
}
Environment (please complete the following information):
- latest dataprepper docker image (2.6.1)