[Rule Tuning] Update rules using NPC integration and non-ECS fields

[brokensound77]

Issues

related to elastic/dev/issues/2270

Summary

The NPC integration schema was updated to move non-ecs fields out of the root namespace. This had minimal impact, only requiring changes in 4 rules.

Rules needing to go from type -> network.protocol:

• 11013227-0301-4a8c-b150-4db924484475 Abnormally Large DNS Response
• cf53f532-9cc9-445a-9ae7-fced307ec53c Cobalt Strike Command and Control Beacon
• 4a4e23cf-78a2-449c-bac3-701924c269d3 Possible FIN7 DGA Command and Control Behavior

Rule needing to go from status -> http.response.status_phrase:

• 31295df3-277b-4c56-a1fb-84e31b4222a9 Inbound Connection to an Unsecure Elasticsearch Node

I also converted the Lucene rules to EQL or KQL for maintainability purposes.

Details

cf53f532-9cc9-445a-9ae7-fced307ec53c Cobalt Strike Command and Control Beacon

• also converted to EQL

4a4e23cf-78a2-449c-bac3-701924c269d3 Possible FIN7 DGA Command and Control Behavior

• also converted to EQL

31295df3-277b-4c56-a1fb-84e31b4222a9 Inbound Connection to an Unsecure Elasticsearch Node

• also converted to KQL

Problems

As a reminder, Lucene rules are not parsed and queries validated, so some of this has existed, but is just now appearing due to the conversion to KQL

http.response.status_phrase

This field exists in the beats and integration custom schemas, but not ECS. The logic however, may be creating issues with how the extended schemas are validating and so may need revision.

Error at line:2,column:5
Unknown field
(event.dataset:network_traffic.http or (event.category:network_traffic and network.protocol:http)) and
    http.response.status_phrase:ok and destination.port:9200 and network.direction:inbound and
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^
Try adding event.module or event.dataset to specify beats module
stack: 8.12.0, beats: 8.10.3, ecs: 8.10.0

http.response.headers.content-type and http.request.headers.authorization

Header fields (http.response.headers.*) do not exist in ECS, but they do exist as an object under the integration and beats custom schemas, however, I do not think that they are being built properly when the schemas are downloaded an assembled. I assume this is because the type is object, which should imply * wildcard fields if not further defined. We need to explore that.

• Under integration, the type is flattened
• Under beats, the assembled schema is missing the field altogether, even though it exists in the packetbeat http module

Error at line:3,column:9
Unknown field
(event.dataset:network_traffic.http or (event.category:network_traffic and network.protocol:http)) and
    http.response.status_phrase:ok and destination.port:9200 and network.direction:inbound and
    not http.response.headers.content-type:"image/x-icon" and
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
	Try adding event.module or event.dataset to specify integration module
	Will check against integrations ['network_traffic'] combined.
	package='network_traffic', integration='http', package_version='1.25.1', stack_version='8.10.0', ecs_version='8.10.0'

Ultimately, we may be able to (or need to) simplify the logic for the query, but we also need to solve the dynamic schema from object type as well

[brokensound77]

After chatting more with @efd6, the header.* fields will also be updated to be more compatible as:

• http.response.Headers.*
• http.request.Headers.*

We can await that upstream change, pull in the integrations updates, and allow that to pass, however, we still need to look into whether schema of type object is built properly in beats and integrations

[botelastic[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[w0rk3r]

@brokensound77, is this one ready to proceed, as the related PRs/Issues seem to be resolved?

[brokensound77]

Due to this bug, this is blocked until #3556 is resolved!