kibana icon indicating copy to clipboard operation
kibana copied to clipboard
verified publisher icon elastic

[ResponseOps] Cases analytics index

Open adcoelho opened this issue 6 months ago • 1 comments

Closes #221234

Merging into a feature branch

Summary

This PR adds:

  • The analytics index creation when the cases plugin starts.
  • Backfill task registration.
  • Backfill task scheduling.
  • Mapping version 1 for:
    • .internal.cases
    • .internal.cases-comments
    • .internal.cases-attachments
  • All corresponding painless scripts.

How to review

  1. The main functionality is in analytics_index.ts. The class in this file implements the index creation algorithm of the RFC.
  2. Three folders instantiate this class to create the indexes we need:
  • cases_analytics/cases_index/
  • cases_analytics/comments_index/
  • cases_analytics/attachments_index/
  1. The backfill logic can be found in cases_analytics/tasks/backfill_task.
  2. I moved the retry_service previously used by the cases connector to a common folder and started using it in the AnalyticsIndex class.
  3. The call to create the indexes is done in x-pack/platform/plugins/shared/cases/server/plugin.ts.

How to test

A few things are needed.

  1. Create a role that has index creation permission. Below is what works for me; it might be overkill. Screenshot 2025-04-25 at 11 30 45

  2. Create a user with the role created in step 1. Let's call the user index_creator. The password should be changeme. Screenshot 2025-04-25 at 11 31 27

  3. Start Kibana with this user as the Elasticsearch user - yarn start with the option --elasticsearch.username=index_creator.

  4. Kibana startup logs should show relevant messages:

[INFO ][plugins.cases] [.internal.cases] Checking if index exists.
[INFO ][plugins.cases] [.internal.cases-attachments] Checking if index exists.
[INFO ][plugins.cases] [.internal.cases-comments] Checking if index exists.
  1. You can check the content of these indexes in the Dev Tools.
GET /.internal.cases/_search
GET /.internal.cases-comments/_search
GET /.internal.cases-attachments/_search
  1. You can check the mapping of each of these indexes in the Dev Tools.
GET /.internal.cases
GET /.internal.cases-comments
GET /.internal.cases-attachments
  1. You can check the painless scripts in the Dev Tools.
GET /_scripts/cai_cases_script_1
GET /_scripts/cai_attachments_script_1
GET /_scripts/cai_comments_script_1

Summary

Summarize your PR. If it involves visual changes include a screenshot or gif.

Checklist

Check the PR satisfies following conditions.

Reviewers should verify this PR satisfies this list as well.

  • [ ] Any text added follows EUI's writing guidelines, uses sentence case text and includes i18n support
  • [ ] Documentation was added for features that require explanation or tutorials
  • [ ] Unit or functional tests were updated or added to match the most common scenarios
  • [ ] If a plugin configuration key changed, check if it needs to be allowlisted in the cloud and added to the docker list
  • [ ] This was checked for breaking HTTP API changes, and any breaking changes have been approved by the breaking-change committee. The release_note:breaking label should be applied in these situations.
  • [ ] Flaky Test Runner was used on any tests changed
  • [ ] The PR description includes the appropriate Release Notes section, and the correct release_note:* label is applied per the guidelines

Identify risks

Does this PR introduce any risks? For example, consider risks like hard to test bugs, performance regression, potential of data loss.

Describe the risk, its severity, and mitigation for each identified risk. Invite stakeholders and evaluate how to proceed before merging.

adcoelho avatar Jun 11 '25 13:06 adcoelho

Pinging @elastic/response-ops (Team:ResponseOps)

elasticmachine avatar Jun 17 '25 08:06 elasticmachine

Pinging @elastic/response-ops-cases (Feature:Cases)

elasticmachine avatar Jun 17 '25 08:06 elasticmachine

I am testing the PR with one million cases. As I test, I will post any issues/bugs I find, if any. So far I have found two small bugs:

  • The @timestamp is not populated in the activity index.
  • The create_case user action activity is copied to the activity index. Based on the painless script, this should not happen. Also, I think we should not copy them because there will be one for each case, and for the initial phase, maybe this is too much.

cnasikas avatar Jun 20 '25 14:06 cnasikas

@elasticmachine merge upstream

cnasikas avatar Jun 23 '25 06:06 cnasikas

I did the following tests and everything is working as expected 🎉 !! In all tests, I had 1M cases and 10 rules running with a case action that creates new cases and attaches alerts to them every 5 minutes. I also had 4 Kibana nodes run in parallel.

  • Start with no indices and backfill 1M cases. The backfill ran smoothly, and new cases were populated correctly. Mappings and scripts were correct.
  • No errors with mappings or scripts.
  • All fields were populated correctly.
  • I updated the mappings, the scripts, and increased the version. The new field was populated correctly in all cases.
  • I deleted the indices, and the sync task started to fail as expected. Kibana was not affected.
  • All newly created or updated cases, along with their corresponding saved objects, were populated to the indices correctly.
  • Errors on startup did not affect Kibana startup.
  • I created a dashboard and data views to visualize the data. They were never affected by the process.
Screenshot 2025-06-23 at 11 47 17 AM

cnasikas avatar Jun 23 '25 08:06 cnasikas

Also, I noticed that when the reindex fails, due to painless script errors, it updates the last success time to be the last attempt even though an error happened.

Fixed in https://github.com/elastic/kibana/pull/223405/commits/947cd802a19016584ead580d8b75543ab8740fac

cnasikas avatar Jun 23 '25 16:06 cnasikas

The @timestamp is not populated in the activity index.

Fixed.

The create_case user action activity is copied to the activity index.

Fixed in 27dc400 by whitelisting the types we want to see.

I also changed the default of the config var to true

adcoelho avatar Jun 23 '25 18:06 adcoelho

:green_heart: Build Succeeded

Metrics [docs]

✅ unchanged

History

  • :green_heart: Build #311606 succeeded 947cd802a19016584ead580d8b75543ab8740fac
  • :green_heart: Build #311235 succeeded ee2a3db63388e9dae4f751d9e2d8b05d28c3ddea
  • :green_heart: Build #310947 succeeded 351735780aa82dfedfd47818d30b135373d23109
  • :green_heart: Build #310725 succeeded 5d42ded2ea17601f5866be2cd312e3dbf5193be3
  • :green_heart: Build #309330 succeeded 35deb011c3ca331e20ea39463fb2360f7233a5f2

cc @adcoelho

elasticmachine avatar Jun 23 '25 20:06 elasticmachine

Starting backport for target branches: 8.19

https://github.com/elastic/kibana/actions/runs/15843332016

kibanamachine avatar Jun 24 '25 06:06 kibanamachine

💔 All backports failed

Status Branch Result
❌ 8.19 Backport failed because of merge conflicts

Manual backport

To create the backport manually run:

node scripts/backport --pr 223405

Questions ?

Please refer to the Backport tool documentation

kibanamachine avatar Jun 24 '25 06:06 kibanamachine

💚 All backports created successfully

Status Branch Result
✅ 8.19

Note: Successful backport PRs will be merged automatically after passing CI.

Questions ?

Please refer to the Backport tool documentation

adcoelho avatar Jun 24 '25 08:06 adcoelho