kibana
kibana copied to clipboard
elastic
Auto generate ECS fields
- Auto generate ECS fields from ECS 8.5 branch
- Increase field limits for indexes as we did previously here
How I did generate:
- Download ecs repo from 8.5 branch
- rename folder to ecs
-
cd ${KIBANA_FOLDER} -
cd .. - move/past
ecsfoder here -
cd kibana -
node x-pack/plugins/rule_registry/scripts/generate_ecs_fieldmap/index.js
I'm not seeing any version labels - what stack release are we targetting for this?
I'm not seeing any version labels - what stack release are we targetting for this?
@pmuellr The goal is to merge it for 8.5, but the branch not yet created, so I didn't add any specific label
Do we actually need and benefit from all of these ECS fields added to the existing alerts-as-data indices? Prior to us minimizing the number of alert indices that are created as outlined by the future of alerts as data, we should be careful about adding new fields to the alerts-as-data indices. Each field we add to the mappings will increase Elasticsearch's data node heap usage.
This isn't as bad for customers who don't have a large number of Kibana spaces, adding more fields isn't "that bad". For example, if the customer only has a single space, increasing from 1,500 to 2,000 fields will only increase the data node heap usage for alerts indices that are storing 12 months of data from 18.43 MB to 24.58 MB. However, if they have a 1,000 spaces this will increase the data node heap usage from 18.43 GB to 24.58 GB.
Do we actually need and benefit from all of these ECS fields added to the existing alerts-as-data indices?
@kobelb that's a complicated question. This PR does incorporate fields that we do not currently use, as this PR includes:
- beta Host Risk fields
- general ECS fields added since previous 8.x PR
I think an argument can be made that the beta fields are more numerous than they should be (due to limitations with the ECS tooling), which I'm currently pursuing. That will limit the fields in the first data point to ones that we DO use.
That still leaves the second question, however: if ECS introduces new fields, and users start adopting those fields in their data, when (if at all) do we start supporting that on the solution side?
For clarity, can someone confirm how many new fields would actually be added to the alert index mappings with this PR?
Here is some additional background:
- ECS 8.5 has 60 more fields than ECS 8.4. (1574 vs 1514)
- Of those 60 new fields, 54 of them are related to the new risk fields (6 new fields x 9 nesting places). As @rylnd mentions, we can likely eliminate some/most of the nesting places with updated ECS tooling to reduce this number further.
- There are 5 other new ECS fields.
- The motivation for this mappings update was driven by the need to add support for the new risk fields to support entity analytics enriching alert documents with entity risk info. We expected that this update would result in adding only the "new in 8.5" ECS fields to the alert mappings, which would have added no more than 60 fields.
The original design principle for the security signals index (now the Alerts index) was that it would always support the full set of ECS fields that were current at the time of release. This was driven by the draft RFC Guidelines for ensuring compatibility between Elastic ECS data producers and consumers #1 which states that the security solution "MUST work properly on any data that is compliant with the latest published version of ECS at the time of the Elastic Stack feature freeze for the app/content."
Over approximately the past year The number of ECS fields had grown as follows:
- 1.11 - 1171
- 1.12 - 1239
- 8.0 - 1269
- 8.1 - 1315
- 8.2 - 1503 <- Around here, out of concern for rapidly growing mapping size, we deprioritized the guideline above.
- 8.3 - 1511
- 8.4 - 1514
- 8.5 - 1574
Since approximately the 8.2 timeframe, we've been in an ambiguous state, agreeing to be intentional about adding new fields to keep the alert index mappings from exploding in size, but at the same time, living with the tension of possible user dissatisfaction because we've made an exception to the ECS "contract" to fully support users' ECS-compliant data in solutions (i.e. if user data is in ECS format, it will be added to detection Alerts.)
In essence, we've created a de facto "exclude list" of ECS fields that are not supported in the Alert field mappings. AFAIK, this list is not documented in user-facing documents, and there seems to be some lack of clarity as to how to update alert index mappings with some new ECS fields, but not all ECS fields.
At this point, there seems to be two options:
-
(default) Continue with the agreement to be intentional about adding fields to the Alert mappings PRO: limits the growth of mapping size of Alerts index. CON: weakens ECS contract with users, causes dev confusion that could result in mapping conflicts or breaking changes for users. Action: Modify the automation in this PR to exclude previously excluded fields. Define process for updating, maintaining, documenting this exclusion list.
-
Overturn our 8.2-timeframe decision, and again adhere to the principle of including all ECS fields in the Alert index mappings. PRO: Users enjoy the confidence that if their data uses ECS, their fields will get included in Alerts. Much simpler process for solution dev teams - every release, include all the latest ECS mappings. CON: Alerts index mappings grow at the same rate as ECS. Action: Continue with the automation in this PR to add all ECS fields.
@MikePaquette This script generate a new 273 fields:
Show fields
{
"client": {
"user": {
"risk": {
"calculated_level": {},
"calculated_score": {},
"calculated_score_norm": {},
"static_level": {},
"static_score": {},
"static_score_norm": {}
}
}
},
"container": {
"cpu": {
"usage": {}
},
"disk": {
"read": {
"bytes": {}
},
"write": {
"bytes": {}
}
},
"image": {
"hash": {
"all": {}
}
},
"memory": {
"usage": {}
},
"network": {
"egress": {
"bytes": {}
},
"ingress": {
"bytes": {}
}
}
},
"destination": {
"user": {
"risk": {
"calculated_level": {},
"calculated_score": {},
"calculated_score_norm": {},
"static_level": {},
"static_score": {},
"static_score_norm": {}
}
}
},
"dll": {
"hash": {
"sha384": {},
"tlsh": {}
},
"pe": {
"pehash": {}
}
},
"email": {
"attachments": {
"file": {
"extension": {},
"hash": {
"md5": {},
"sha1": {},
"sha256": {},
"sha384": {},
"sha512": {},
"ssdeep": {},
"tlsh": {}
},
"mime_type": {},
"name": {},
"size": {}
}
},
"bcc": {
"address": {}
},
"cc": {
"address": {}
},
"content_type": {},
"delivery_timestamp": {},
"direction": {},
"from": {
"address": {}
},
"local_id": {},
"message_id": {},
"origination_timestamp": {},
"reply_to": {
"address": {}
},
"sender": {
"address": {}
},
"subject": {},
"to": {
"address": {}
},
"x_mailer": {}
},
"faas": {
"id": {},
"name": {},
"version": {}
},
"file": {
"hash": {
"sha384": {},
"tlsh": {}
},
"pe": {
"pehash": {}
}
},
"host": {
"boot": {
"id": {}
},
"pid_ns_ino": {},
"risk": {
"calculated_level": {},
"calculated_score": {},
"calculated_score_norm": {},
"static_level": {},
"static_score": {},
"static_score_norm": {}
}
},
"log": {
"syslog": {
"appname": {},
"hostname": {},
"msgid": {},
"procid": {},
"structured_data": {},
"version": {}
}
},
"orchestrator": {
"cluster": {
"id": {}
},
"resource": {
"id": {},
"ip": {},
"parent": {
"type": {}
}
}
},
"process": {
"entry_leader": {
"args": {},
"args_count": {},
"attested_groups": {
"name": {}
},
"attested_user": {
"id": {},
"name": {}
},
"command_line": {},
"entry_meta": {
"source": {
"ip": {}
},
"type": {}
},
"executable": {},
"group": {
"id": {},
"name": {}
},
"interactive": {},
"name": {},
"parent": {
"entity_id": {},
"pid": {},
"session_leader": {
"entity_id": {},
"pid": {},
"start": {}
},
"start": {}
},
"pid": {},
"real_group": {
"id": {},
"name": {}
},
"real_user": {
"id": {},
"name": {}
},
"same_as_process": {},
"saved_group": {
"id": {},
"name": {}
},
"saved_user": {
"id": {},
"name": {}
},
"start": {},
"supplemental_groups": {
"id": {},
"name": {}
},
"tty": {
"char_device": {
"major": {},
"minor": {}
}
},
"user": {
"id": {},
"name": {}
},
"working_directory": {}
},
"env_vars": {},
"group_leader": {
"args": {},
"args_count": {},
"command_line": {},
"entity_id": {},
"executable": {},
"group": {
"id": {},
"name": {}
},
"interactive": {},
"name": {},
"pid": {},
"real_group": {
"id": {},
"name": {}
},
"real_user": {
"id": {},
"name": {}
},
"same_as_process": {},
"saved_group": {
"id": {},
"name": {}
},
"saved_user": {
"id": {},
"name": {}
},
"start": {},
"supplemental_groups": {
"id": {},
"name": {}
},
"tty": {
"char_device": {
"major": {},
"minor": {}
}
},
"user": {
"id": {},
"name": {}
},
"working_directory": {}
},
"hash": {
"sha384": {},
"tlsh": {}
},
"interactive": {},
"parent": {
"group": {
"id": {},
"name": {}
},
"group_leader": {
"entity_id": {},
"pid": {},
"start": {}
},
"hash": {
"sha384": {},
"tlsh": {}
},
"interactive": {},
"pe": {
"pehash": {}
},
"real_group": {
"id": {},
"name": {}
},
"real_user": {
"id": {},
"name": {}
},
"saved_group": {
"id": {},
"name": {}
},
"saved_user": {
"id": {},
"name": {}
},
"supplemental_groups": {
"id": {},
"name": {}
},
"tty": {
"char_device": {
"major": {},
"minor": {}
}
},
"user": {
"id": {},
"name": {}
}
},
"pe": {
"pehash": {}
},
"previous": {
"args": {},
"args_count": {},
"executable": {}
},
"real_group": {
"id": {},
"name": {}
},
"real_user": {
"id": {},
"name": {}
},
"saved_group": {
"id": {},
"name": {}
},
"saved_user": {
"id": {},
"name": {}
},
"session_leader": {
"args": {},
"args_count": {},
"command_line": {},
"executable": {},
"group": {
"id": {},
"name": {}
},
"interactive": {},
"name": {},
"parent": {
"entity_id": {},
"pid": {},
"session_leader": {
"entity_id": {},
"pid": {},
"start": {}
},
"start": {}
},
"pid": {},
"real_group": {
"id": {},
"name": {}
},
"real_user": {
"id": {},
"name": {}
},
"same_as_process": {},
"saved_group": {
"id": {},
"name": {}
},
"saved_user": {
"id": {},
"name": {}
},
"start": {},
"supplemental_groups": {
"id": {},
"name": {}
},
"tty": {
"char_device": {
"major": {},
"minor": {}
}
},
"user": {
"id": {},
"name": {}
},
"working_directory": {}
},
"supplemental_groups": {
"id": {},
"name": {}
},
"tty": {
"char_device": {
"major": {},
"minor": {}
},
"columns": {},
"rows": {}
},
"user": {
"id": {},
"name": {}
}
},
"server": {
"user": {
"risk": {
"calculated_level": {},
"calculated_score": {},
"calculated_score_norm": {},
"static_level": {},
"static_score": {},
"static_score_norm": {}
}
}
},
"service": {
"node": {
"role": {},
"roles": {}
},
"origin": {
"node": {
"role": {},
"roles": {}
}
},
"target": {
"node": {
"role": {},
"roles": {}
}
}
},
"source": {
"user": {
"risk": {
"calculated_level": {},
"calculated_score": {},
"calculated_score_norm": {},
"static_level": {},
"static_score": {},
"static_score_norm": {}
}
}
},
"threat": {
"enrichments": {
"indicator": {
"file": {
"hash": {
"sha384": {},
"tlsh": {}
},
"pe": {
"pehash": {}
}
}
},
"matched": {
"occurred": {}
}
},
"feed": {
"dashboard_id": {},
"description": {},
"name": {},
"reference": {}
},
"indicator": {
"file": {
"hash": {
"sha384": {},
"tlsh": {}
},
"pe": {
"pehash": {}
}
}
}
},
"user": {
"changes": {
"risk": {
"calculated_level": {},
"calculated_score": {},
"calculated_score_norm": {},
"static_level": {},
"static_score": {},
"static_score_norm": {}
}
},
"effective": {
"risk": {
"calculated_level": {},
"calculated_score": {},
"calculated_score_norm": {},
"static_level": {},
"static_score": {},
"static_score_norm": {}
}
},
"risk": {
"calculated_level": {},
"calculated_score": {},
"calculated_score_norm": {},
"static_level": {},
"static_score": {},
"static_score_norm": {}
},
"target": {
"risk": {
"calculated_level": {},
"calculated_score": {},
"calculated_score_norm": {},
"static_level": {},
"static_score": {},
"static_score_norm": {}
}
}
}
}
Decision:
-
Short term: Add only the 12 new risk fields (6 fields x 2 nesting places [
host.*,user.*]) to the Alert index mapptings for 8.5 - Next: Option 1 - Modify the automation in this PR to exclude previously excluded fields. Define process for updating, maintaining, documenting this exclusion list, and for adding selected new ECS fields.
If end up doing a subset of fields, at a minimum we need:
-
host.* -
cloud.* -
orchestrator.* -
labels.* -
tags
If end up doing a subset of fields, at a minimum we need:
@simianhacker is this specific to the risk.* nestings, or are you talking generally about observability's field needs (relative to what's missing from current mappings)?
More explicitly: are you saying that there are NEW ECS fields that o11y needs in 8.5, or are you just requesting that we not remove any of the existing fields?
@rylnd I'm saying at a minimum, O11y needs those "top level" fields (objects) added to the mappings for Alert-As-Data.
I updated PR, with only risk fields and some updates for fields that @simianhacker mention
:green_heart: Build Succeeded
- Buildkite Build
- Commit: 54b8f2d2761f9f6d0a08ff1bddce1cc84229465a
Metrics [docs]
Async chunks
Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app
| id | before | after | diff |
|---|---|---|---|
observability |
525.4KB | 526.5KB | +1.1KB |
securitySolution |
6.6MB | 6.6MB | +1.1KB |
triggersActionsUi |
1.1MB | 1.1MB | +1.1KB |
| total | +3.4KB |
History
- :broken_heart: Build #75150 failed 7032a3eab78f6f8dad58b53d52ad2d771b009165
- :green_heart: Build #74567 succeeded e5660c0c010c2c67f8dc3e7c94fc60d8597d7faa
- :yellow_heart: Build #74441 was flaky cd06cb27bb9c57679182874c3a4af63646bad4e9
- :broken_heart: Build #74322 failed 3f81ab7750e279ef3938533a7ebe9ea8b402d033
- :broken_heart: Build #74141 failed 76e2d93e8e8dc8b3ea8d3533eab7e28f0f08ff9d
To update your PR or re-run it, just comment with:
@elasticmachine merge upstream
💚 All backports created successfully
| Status | Branch | Result |
|---|---|---|
| ✅ | 8.5 |  |
Note: Successful backport PRs will be merged automatically after passing CI.
Questions ?
Please refer to the Backport tool documentation
