kibana Implement functionality to add observables, procedures and custom fields to alerts for TheHive

Summary

Added a toggle to retain the severity from the rule. When enabled, alerts generated from the rule will inherit its severity; otherwise, users must manually select a severity level from the dropdown.
Added a template selection menu with predefined basic templates. These templates come with preset configurations, including observables and procedures, which automatically populate the Body field upon selection. Users also have the option to modify an existing template or create a custom one using the Custom Template option.

Screenshots

image (35) image (36)

Checklist

[x] Any text added follows EUI's writing guidelines, uses sentence case text and includes i18n support
[x] Documentation was added for features that require explanation or tutorials
[x] Unit or functional tests were updated or added to match the most common scenarios
[x] If a plugin configuration key changed, check if it needs to be allowlisted in the cloud and added to the docker list
[x] This was checked for breaking HTTP API changes, and any breaking changes have been approved by the breaking-change committee. The release_note:breaking label should be applied in these situations.
[x] Flaky Test Runner was used on any tests changed
[ ] The PR description includes the appropriate Release Notes section, and the correct release_note:* label is applied per the guidelines

Jan 20 '25 18:01 brijesh-elastic

A documentation preview will be available soon.

🔨 Buildkite builds
📚 HTML diff
📙 Preview page

Request a new doc build by commenting

Rebuild this PR: run docs-build
Rebuild this PR and all Elastic docs: run docs-build rebuild

_{run docs-build is much faster than run docs-build rebuild. A rebuild should only be needed in rare situations.}

_{If your PR continues to fail for an unknown reason, the doc build pipeline may be broken. Elastic employees can check the pipeline status here.}

Jan 20 '25 18:01 github-actions[bot]

Pinging @elastic/response-ops (Team:ResponseOps)

Jan 21 '25 06:01 elasticmachine

Pinging @elastic/response-ops-cases (Feature:Cases)

Jan 21 '25 06:01 elasticmachine

Shouldn't this also be reviewed by someone in security? I thought we do not own this connector

Jan 27 '25 13:01 jcger

Based on. our conversation here: https://github.com/elastic/kibana/pull/207255#discussion_r1934520482. I realized that the backend accepts a template, but it does not do something with it. The schema is defined as:

template: schema.nullable(schema.string({ defaultValue: TheHiveTemplate.BUILD_YOUR_OWN })),
body: schema.nullable(schema.string()),

I can define a template, a body, or both. If I use the API (outside of the UI) and the template field, nothing will happen on the backend. It seems the template is used only for the UI. Also, TheHiveTemplate.BUILD_YOUR_OWN results in Build Your Own, which is not a friendly value for the backend. We can remove the template from the schema, as it is not used by the backend, and accept only the body and change the UI like (along with the context variables or in a different button on the left and without the filter search bar. We need design feedback):

This way, we do not have to know the template. The users selected one of the templates is being populated in the JSON editor.

Jan 30 '25 09:01 cnasikas

Feb 12 '25 17:02 github-actions[bot]

Feb 13 '25 12:02 github-actions[bot]

Feb 13 '25 13:02 github-actions[bot]

Feb 18 '25 10:02 github-actions[bot]

Feb 18 '25 10:02 github-actions[bot]

Feb 20 '25 10:02 github-actions[bot]

Feb 20 '25 10:02 github-actions[bot]

@cnasikas is this review currently with ResponseOps or waiting on a review from @asnehalb team in security?

Apr 22 '25 14:04 jamiehynds

@jamiehynds There are unresolved comments from the initial review, and the intermediate release process for Serverless needs to be followed. The Security team was informed at the time to help with the process and the feedback.

Apr 22 '25 14:04 cnasikas

Thanks!

@brijesh-elastic are you ok to address any unresolved comments from the initial review?

@asnehalb once those comments are addressed, are your team ok to review prior to the Response Ops review?

Apr 22 '25 15:04 jamiehynds

@jamiehynds - we have already resolved all comments, @brijesh-elastic - I could see Merge Conflict kindly resolve it and update the PR please.

Apr 23 '25 07:04 piyush-elastic

Thanks @piyush-elastic! @jamiehynds @asnehalb The intermediate release process for Serverless needs to be followed in this PR. The Security team needs to help @brijesh-elastic with that. Please reach out to us internally to guide you. @semd is familiar with the process.

Apr 23 '25 07:04 cnasikas

@jamiehynds yes, I plan to review it this week.

@cnasikas We can help, but we are not familiar with the intermediate release process; this is the first time we have to follow it.

cc @asnehalb

May 06 '25 11:05 semd

@semd Let's talk offline on how you can achieve that. We can provide documentation that I can share.

May 06 '25 19:05 cnasikas

@semd @brijesh-elastic are we on track to get this PR reviewed and hopefully merged prior to 9.1 FF?

May 28 '25 11:05 jamiehynds

@brijesh-elastic We have been testing the PR. In general everything works well 👍 However, we had a problem with a template:

When using the Compromised User Account Investigation template, the Hive API was throwing an error when the data was sent, the reason is the procedures, what we were sending is:

  "procedures": [
    {
      "patternId": "",
      "occurDate": 1748429891332
    }
  ],

In our rule, the "patternId": "{{#context.alerts}}{{threat.technique.id}}{{/context.alerts}}" evaluates to an empty string, and the TheHIve API does not consider the JSON body as invalid:

Status code: 400. Message: API Error: BadRequest - Invalid json

We tried hardcoding the patternId with an arbitrary number, and the response was:

Status code: 404. Message: API Error: NotFoundError - Pattern not found

When we removed the procedures entirely, the execution worked fine.

Not sure how we can make the threat.technique.id field get populated, but we should make these templates more resilient, it will be hard to debug if a user gets into the same problem. Maybe we could hide this specific template if we know it is going to fail, or add an if (moustache syntax) for the "procedures" entry in the template.

wdyt?

May 28 '25 11:05 semd

@jamiehynds I have been testing it today, and it looks great in general. Only one problem was spotted (see previous comment). We should be able to merge before FF if we work it out quickly.

May 28 '25 11:05 semd

@semd, Regarding the comment , to address the issue of invalid JSON, we can modify the template. During implementation, we identified that threat.technique.id is an array, which requires repeating the object and adding a comma after each — except the last one — to maintain valid JSON. As we have to hardcode a comma after every object, this resulted in invalid JSON due to an extra comma at the end. As a workaround, we can add an extra object same as the first object, but due to this it will show duplicate TTPs in thehive.

"procedures": [
    {{#context.alerts}}
    {{#[threat.technique.id](http://threat.technique.id/)}}
    {
      "patternId": "{{.}}",
      "occurDate": {{#FormatDate}}{{signal.original_time}} ;; x {{/FormatDate}}
    },
    {{/[threat.technique.id](http://threat.technique.id/)}}
    {
      "patternId": "{{context.alerts.0.threat.technique.id.0}}",
      "occurDate": {{#FormatDate}}{{signal.original_time}} ;; x {{/FormatDate}}
    }
    {{/context.alerts}}
  ]

Do you consider this workaround acceptable to continue supporting the template, or would you prefer that we remove the template altogether?

May 29 '25 12:05 brijesh-elastic

Hey @brijesh-elastic, I understand your reasoning, but this logic is becoming quite complex for a prebuilt template. Considering we have the option to introduce a custom body, I would not include this one. So we keep prebuilt templates simple. If users want to have a more advanced body, like this one, they can still do it.

Could you do the change?

May 29 '25 13:05 semd

@elasticmachine merge upstream

Jun 05 '25 14:06 cnasikas

I noticed there are some existing function tests for this connector. It would be good to add more to test the new functionality, if possible.

https://github.com/elastic/kibana/blob/main/x-pack/test/alerting_api_integration/security_and_spaces/group2/tests/actions/connector_types/thehive.ts

Jun 06 '25 20:06 pmuellr

Discussed with @cnasikas and @semd, decided to remove custom templates feature for this release.

Jun 19 '25 11:06 brijesh-elastic

Yes, we'll add.

Jun 19 '25 12:06 brijesh-elastic

:yellow_heart: Build succeeded, but was flaky

Buildkite Build
Commit: 46492e6c07a3f7bafe0de1188d760b9b5ea73788

Failed CI Steps

FTR Configs #38

Test Failures

[job] [logs] FTR Configs #38 / core route timeouts idle socket should timeout if payload sending has too long of an idle period

Metrics [docs]

Async chunks

id	before	after	diff
`stackConnectors`	624.9KB	627.8KB	+2.9KB

History

:green_heart: Build #310252 succeeded 80644a2cd458cca6ddbfe972045d1385303b66d8
:yellow_heart: Build #309896 was flaky 4d664bd4401f7d3ae4c13c6d7409c3cfaefa3c42
:yellow_heart: Build #309370 was flaky 6c7cc74fbb1748ea6888f4215efe1a0eee0488f4
:broken_heart: Build #308904 failed 2c84bbe4c0a9b1b9393d1538af0e43286732c3f7
:yellow_heart: Build #305851 was flaky d394209ad3c04700851a850cb65d967b1a3c0bdc
:green_heart: Build #303948 succeeded 4d8dcbe4aec7cac69b2c4a24cd441b09ab45e987

Jun 19 '25 13:06 elasticmachine

Starting backport for target branches: 8.19

https://github.com/elastic/kibana/actions/runs/15759950221

Jun 19 '25 14:06 kibanamachine