[Security Solution] Security AI Assistant persistent storage.

[YulNaumenko]

Summary

This PR including both new APIs and client side changes to use data stream for Security Solution AI Assistant conversations persistence storage. Issue https://github.com/elastic/security-team/issues/7810

Extended description

elastic-assistant plugin

All API changes are introduced in elastic-assistant plugin server

• RequestContextFactory - this class helps to provide the needed context for each API request for routes handler context.

• AIAssistantService - This service instance is created on the plugin setup and included to the request context factory. It is responsible for the needed conversations storage resources initialization and installation. It uses DataStreamAdapter from packages/kbn-data-stream-adapter. Conversations fieldMap definition here

• AIAssistantConversationsDataClient - data client which has a set of methods to interact with conversation storage on behalf of the current user and space.

• ConversationDataWriter - is a helper class which implements a bulk method to interact with esClient

• Added new routes using versioned router and openAPI code generator schemas: createConversationRoute readConversationRoute updateConversationRoute deleteConversationRoute appendMessagesRoute findUserConversationsRoute bulkActionsRoute

• Migrated existing knowledge_base, evaluate and post_actions_connector_execute routes to versioned routing and openAPI code generator schemas.

kbn-elastic-assistant package

• removed local storage persistency logic for assistantConversations.
• added API requests definition to communicate to server side.

kbn-elastic-assistant-common package

• Changed transformsRowData function to use async add replacements API.
• Exposed routing URLs with constants file to be available for server and client.

security_solution plugin

• Added migrateConversationsFromLocalStorage for existing conversations in the local storage. This migration happening only for the first time when user doesn't have any conversations persisted in the current space. After mirgation complete, the old local storage key securitySolution.assistantConversation will be removed.
• Passing security related baseConversation as a property to ElasticAssistantProvider
• Changed useAssistantTelemetry to fetch information about the conversation from the conversations API
• Modified useConversationStore to fetch the data from the conversations API /api/elastic_assistant/conversations/current_user/_find and merge with security predefined baseConversations if they are not used(persisted) yet.
• Extracted AssistantTab to a separate lazy loaded file to avoid unnecessary rendering/requests till this tab will be shown in Timeline.

[elasticmachine]

Pinging @elastic/security-threat-hunting (Team:Threat Hunting)

[andrew-goldstein]

The following 400 Bad Request status code for PUT requests was observed (via Chrome dev tools) as part of the response to a prompt: Which alerts should I look at first?

PUT /api/elastic_assistant/conversations/52d69752-b9c0-4356-aabf-3f031c088b52 HTTP/1.1

Payload:

{
   "id":"52d69752-b9c0-4356-aabf-3f031c088b52",
   "replacements":{
      "def96100-aaa7-45ad-9391-8dab88f19cfe":"a10a4aa921991f68d3730e2d9c3dd516c90b2ad95093762cfd6e6dddb261532c",
      "31ef44bf-4f48-448e-868e-562446e2ba76":"Host-ttcray5sob",
      "5b469eea-6f56-406b-88ea-c384ac52617b":"3e6eaea61e122bf9bc78a84b1af3614d2397e304785caa056a69d8ef3c6084ff",
      "82abd8d8-2f74-4c89-bd10-b1f7b9923c07":"7cfcc1a3e4487a8087e4ad8877915b7ef5ebb9bb0bb4845ccb57ab4ca642a073",
      "b9737f71-ca58-44f0-9f1e-c5f6da9b59b0":"nh9b9x4mfk",
      "ed9a1373-6bd5-4f4e-9d62-a5bccfd696bc":"5wze689uxw",
      "eab62ab9-5fa8-4a6e-9a5f-9e73a33505e6":"34f9a3c947d202313edc25d4f66f126c8d10f55387a821c8d988c1df8e59e3bc",
      "b59105ae-86b6-4b79-9618-b42073b6d00a":"829n1v5cqy",
      "7247ee36-5b2d-41bf-8b4b-69f9a7606bbd":"4ccvijl6pn",
      "1a84a7e6-63c5-4517-a2e8-859bebf90ec9":"3f36f9c4435da834362aef4705e78e9feb36de5b65a7909986aee551afcb560e",
      "6773f474-3f4a-4d33-b723-3d426db7b633":"52ohclx2su",
      "8797378b-a717-4197-8771-452cd4fa2fb0":"kz044fpr1z",
      "8b26a774-f466-4b37-a18a-3a9a099303a6":"fdaa59e9275baa2c477ebd0248d2fe0824b7ac0f37d7f5a7f635bb3c7bda886e",
      "caaa60d0-2992-4648-80a0-8ad2d6b5fac5":"sl5zb0844i",
      "87eab814-87af-40be-bfa2-d25c8f11a27f":"nm1aabowxf",
      "fa5b6b73-1910-4ef9-a749-2be8fbd98d7c":"ca0597f246e577a53ee0e0f3d9de05344e074ecbb809b4f92d36ddf47c6408bf",
      "2f4ab212-ee11-4edd-b66e-474cd1f5b7b6":"1la2pvwh43",
      "241e1670-7951-42e1-8f8d-6f1372163700":"ywtvbplpia",
      "05a8bc25-d12f-4929-ab62-44b928ee3261":"c813c0281ace430e6c986093ed5b63b226eb9d3919b162fa3a2b76e3fd7219bf"
   }
}

{
   "statusCode":400,
   "error":"Bad Request",
   "message":"[request body]: replacements: Expected array, received object"
}

[andrew-goldstein]

POSTs to a conversations messages, e.g.

POST /api/elastic_assistant/conversations/29db0fbe-2599-419d-9d3e-2d8baef5981b/messages HTTP/1.1

made when the streaming feature flag is enabled in config/kibana.dev.yml:

xpack.securitySolution.enableExperimental: ['assistantStreamingEnabled']

and the KB / alerts settings turned off:

return a 400 Bad Request that appears to prevent the assistant from displaying the response, as illustrated by the following example:

POST /api/elastic_assistant/conversations/29db0fbe-2599-419d-9d3e-2d8baef5981b/messages HTTP/1.1

payload:

{
   "messages":[
      {
         "role":"assistant",
         "reader":{
            
         },
         "timestamp":"1/26/2024, 9:50:50 AM",
         "isError":false
      }
   ]
}

response:

{
    "statusCode": 400,
    "error": "Bad Request",
    "message": "[request body]: messages.0.content: Required, messages.0.reader: Expected string, received object"
}

[YulNaumenko]

The following 400 Bad Request status code for PUT requests was observed (via Chrome dev tools) as part of the response to a prompt: Which alerts should I look at first?

PUT /api/elastic_assistant/conversations/52d69752-b9c0-4356-aabf-3f031c088b52 HTTP/1.1

Payload:

{
   "id":"52d69752-b9c0-4356-aabf-3f031c088b52",
   "replacements":{
      "def96100-aaa7-45ad-9391-8dab88f19cfe":"a10a4aa921991f68d3730e2d9c3dd516c90b2ad95093762cfd6e6dddb261532c",
      "31ef44bf-4f48-448e-868e-562446e2ba76":"Host-ttcray5sob",
      "5b469eea-6f56-406b-88ea-c384ac52617b":"3e6eaea61e122bf9bc78a84b1af3614d2397e304785caa056a69d8ef3c6084ff",
      "82abd8d8-2f74-4c89-bd10-b1f7b9923c07":"7cfcc1a3e4487a8087e4ad8877915b7ef5ebb9bb0bb4845ccb57ab4ca642a073",
      "b9737f71-ca58-44f0-9f1e-c5f6da9b59b0":"nh9b9x4mfk",
      "ed9a1373-6bd5-4f4e-9d62-a5bccfd696bc":"5wze689uxw",
      "eab62ab9-5fa8-4a6e-9a5f-9e73a33505e6":"34f9a3c947d202313edc25d4f66f126c8d10f55387a821c8d988c1df8e59e3bc",
      "b59105ae-86b6-4b79-9618-b42073b6d00a":"829n1v5cqy",
      "7247ee36-5b2d-41bf-8b4b-69f9a7606bbd":"4ccvijl6pn",
      "1a84a7e6-63c5-4517-a2e8-859bebf90ec9":"3f36f9c4435da834362aef4705e78e9feb36de5b65a7909986aee551afcb560e",
      "6773f474-3f4a-4d33-b723-3d426db7b633":"52ohclx2su",
      "8797378b-a717-4197-8771-452cd4fa2fb0":"kz044fpr1z",
      "8b26a774-f466-4b37-a18a-3a9a099303a6":"fdaa59e9275baa2c477ebd0248d2fe0824b7ac0f37d7f5a7f635bb3c7bda886e",
      "caaa60d0-2992-4648-80a0-8ad2d6b5fac5":"sl5zb0844i",
      "87eab814-87af-40be-bfa2-d25c8f11a27f":"nm1aabowxf",
      "fa5b6b73-1910-4ef9-a749-2be8fbd98d7c":"ca0597f246e577a53ee0e0f3d9de05344e074ecbb809b4f92d36ddf47c6408bf",
      "2f4ab212-ee11-4edd-b66e-474cd1f5b7b6":"1la2pvwh43",
      "241e1670-7951-42e1-8f8d-6f1372163700":"ywtvbplpia",
      "05a8bc25-d12f-4929-ab62-44b928ee3261":"c813c0281ace430e6c986093ed5b63b226eb9d3919b162fa3a2b76e3fd7219bf"
   }
}

{
   "statusCode":400,
   "error":"Bad Request",
   "message":"[request body]: replacements: Expected array, received object"
}

resolved by commit https://github.com/elastic/kibana/pull/173487/commits/a1fe21af47c3a120edb7b2b7859f3f24ac48f026

[stephmilovic]

@elasticmachine merge upstream

[stephmilovic]

@elasticmachine merge upstream

[stephmilovic]

@elasticmachine merge upstream

[kibana-ci]

:yellow_heart: Build succeeded, but was flaky

• Buildkite Build
• Commit: c9926e7f1b64d641525c6337f1ba80b17244e523
• Cloud Deployment

Failed CI Steps

• Jest Integration Tests #7

Test Failures

• [job] [logs] Jest Integration Tests #7 / getOutdatedDocumentsQuery creates a query returning the expected documents
• [job] [logs] Jest Integration Tests #7 / Higher version doc conversion #bulkGet returns the documents with the correct shape
• [job] [logs] Jest Integration Tests #7 / Higher version doc conversion #bulkResolve returns the documents with the correct shape
• [job] [logs] Jest Integration Tests #7 / Higher version doc conversion #get returns the documents with the correct shape
• [job] [logs] Jest Integration Tests #7 / Higher version doc conversion #resolve returns the documents with the correct shape
• [job] [logs] Jest Integration Tests #7 / Task priority checks detects tasks with priority definitions
• [job] [logs] Jest Integration Tests #7 / task state validation allow_reading_invalid_state: false should fail the task run when setting allow_reading_invalid_state:false and reading an invalid state
• [job] [logs] Jest Integration Tests #7 / task state validation allow_reading_invalid_state: true should debug log by default when reading an invalid task state
• [job] [logs] Jest Integration Tests #7 / task state validation allow_reading_invalid_state: true should drop unknown fields from the task state
• [job] [logs] Jest Integration Tests #7 / task state validation allow_reading_invalid_state: true should fail to update the task if the task runner returns an unknown property in the state
• [job] [logs] Jest Integration Tests #7 / task state validation allow_reading_invalid_state: true should migrate the task state
• [job] [logs] Jest Integration Tests #7 / ZDT upgrades - introducing a new SO type should support adding the bar type
• [job] [logs] Jest Integration Tests #7 / ZDT upgrades - switching from v2 algorithm when switching from a compatible version it able to re-use a cluster state from the v2 algorithm
• [job] [logs] Jest Integration Tests #7 / ZDT upgrades - switching from v2 algorithm when switching from an incompatible version fails and throws an explicit error
• [job] [logs] Jest Integration Tests #7 / ZDT with v2 compat - recovering from partially migrated state migrates the documents

Metrics [docs]

Module Count

Fewer modules leads to a faster build time

id	before	after	diff
securitySolution	5019	5030	+11

Public APIs missing comments

Total count of every public API that lacks a comment. Target amount is 0. Run node scripts/build_api_docs --plugin [yourplugin] --stats comments for more detailed information.

id	before	after	diff
@kbn/elastic-assistant	84	132	+48
@kbn/elastic-assistant-common	57	207	+150
elasticAssistant	27	31	+4
total			+202

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id	before	after	diff
securitySolution	12.8MB	15.5MB	:warning: +2.7MB

Public APIs missing exports

Total count of every type that is part of your API that should be exported but is not. This will cause broken links in the API documentation system. Target amount is 0. Run node scripts/build_api_docs --plugin [yourplugin] --stats exports for more detailed information.

id	before	after	diff
@kbn/elastic-assistant	6	8	+2
stackConnectors	1	0	-1
total			+1

Page load bundle

Size of the bundles that are downloaded on every page load. Target size is below 100kb

id	before	after	diff
securitySolution	72.0KB	72.0KB	-9.0B

Unknown metric groups

API count

id	before	after	diff
@kbn/elastic-assistant	104	154	+50
@kbn/elastic-assistant-common	59	220	+161
elasticAssistant	41	45	+4
total			+215

ESLint disabled line counts

id	before	after	diff
@kbn/elastic-assistant	19	20	+1
elasticAssistant	13	29	+16
securitySolution	482	485	+3
total			+20

Total ESLint disabled count

id	before	after	diff
@kbn/elastic-assistant	20	21	+1
elasticAssistant	13	29	+16
securitySolution	557	560	+3
total			+20

History

• :broken_heart: Build #197564 failed 0c3b0b7bb82096084cb5c3b50574985a440986c7
• :broken_heart: Build #197535 failed ee7e26925d28aec8cbc3e80e26a7a4fccfe64bb3
• :broken_heart: Build #197497 failed d2bcda56f4f85a61cb50251c634d4e4fef5e8a32
• :yellow_heart: Build #197467 was flaky d25950eef56fb970f14bbcc04bd1ed7304f3a519
• :broken_heart: Build #197335 failed acbf3b7ad6e93e16c15348a37475f6bc30bd505c

To update your PR or re-run it, just comment with: @elasticmachine merge upstream

cc @YulNaumenko