cloudbeat
cloudbeat copied to clipboard
JSON stringify unmapped fields
Motivation
Today we don't map resource.raw
, result.evidence
and result.expected
because we these are sent as part of the source, if we will apply dynamic mappings on these fields we reach to the default max limit of number of fields being mapped and we experienced mapping collision which drops the document from being indexed.
So our solution at this time was not to map them, so we at least be able to present these values to the user because they are part of _source.
However, there could be a better solution for this problem which is to map these fields as text and their value will be a stringified JSON
- It keeps the same behavior from user perspective as it is today
- This is our only use of _source in kibana, which blocks us from using synthetic source which optimizes the storage of these indices
- It will allow the user (and us) to query these fields as text which is an improvement
- In the future, es|ql might support querying these fields with
jq
kind of query language (reference)
Definition of done
- [ ] Add mapping as text to cloud_security_package to the fields mentioned above
- [ ] Cloudbeat should send this values as a valid stringified JSON
Out of scope
Related tasks/epics