[Security Solution] AI Assistant - System prompt move

[stephmilovic]

[!CAUTION] Opening PR to get CI green, but not ready for review as I have not completed prompt eval on this work.

Summary

This PR refactors the system prompt strategy to streamline the assistant's performance and reduce issues arising from outdated system prompts. sers will have a more focused AI Assistant, with the flexibility to add custom prompts as needed.

Key Changes

1. Default System Prompt Removal from Client:

   • Removed the default system prompt from the AI Assistant.
     • Removed default system prompts from the creation flow in security_solution/public/assistant/provider.tsx
     • Excluded any previously created default system prompts by adding filter to the query in elastic_assistant/server/routes/prompts/find_route.ts

2. Default System Prompt Elements Migrated to LangGraph:

   • Key elements of the previous system prompts have been integrated directly into the LangGraph, ensuring that the assistant remains focused on relevant topics without the need for a separate system prompt.
   • Reworked the assistant’s focus to ensure it only responds to inquiries related to security, Elastic security, and security operations. This jailing mechanism is now part of the agent prompts in LangGraph.

3. Moved Custom (User) System Prompt Sending Logic to LangGraph

   • Custom system prompts still exist in the UI. They are no longer sent in the first user message, but instead appended to the default system prompt located in the LangGraph.
   • Once the first message of the conversation has been sent, the system prompt is prepended as a message in the conversation, where the user is "System"
   • Side effect: Since the first user message no longer contains the system prompt, if the user changes the system prompt mid conversation in the assistant settings, that change will take effect immediately without clearing the conversation. For example (see trace):

4. Clearing the system prompt now updates the conversation

   • Previously, when a custom system prompt was cleared in the EmptyConvo UI, it was only cleared from the message about to be sent, not from the conversation. You could "unclear" it to bring it back:

     https://github.com/user-attachments/assets/62d5305f-781b-4505-af10-e60bfe06352c

   • Now, when a custom system prompt was cleared in the EmptyConvo UI, an API call is sent to remove the system prompt from the conversation. This is because we are adding the custom system prompt to the user message, and now we are adding it to the LangGraph system prompt on the server side. So the conversation apiConfig.defaultSystemPromptId needs to stay up to date with the UI so we know whether or not to send the prompt.

     https://github.com/user-attachments/assets/fc3c669e-7d76-4f59-9526-909af2b1dfbc

5. General System Prompt Flow Cleanup

   • Resolved an issue where the conversation would change unexpectedly after hitting "Save" when modifying the system prompts' default conversations. The root cause was that AssistantSettings was expecting the conversationSettings object to be indexed by id, while in some cases, it was indexed by title as a fallback. Additionally, there was an instance in SystemPromptEditor where the code called setConversationSettings and incorrectly keyed the update by title instead of id. To fix this, I made two changes. In AssistantSettings, I modified the code that referenced conversationSettings[cId] to use !Object.values(conversationSettings).some(({ id }) => id === cId) to verify the existence of the conversation. Secondly, in SystemPromptEditor, I updated the code to ensure the object is consistently indexed by id instead of title. These adjustments have resolved the bug, ensuring that conversations remain consistent after saving changes. This is what that bug looked like:

     https://github.com/user-attachments/assets/2043117b-b01a-4c81-b3ae-f08e4d6a496d

   • Resolved an issue preventing a conversation's apiConfig.defaultSystemPromptId from being set to undefined after it had been assigned a value. The getUpdateScript function used by the bulk actions route only targeted updates where default_system_prompt_id had a defined value, overlooking cases where it was undefined. By adding a condition to remove the property when it is undefined, this bug was fixed. This is what that bug looked like:

     https://github.com/user-attachments/assets/a0048e46-a09c-4091-a611-c67f7ef6c8fa

[elasticmachine]

Pinging @elastic/security-solution (Team: SecuritySolution)

[stephmilovic]

@elasticmachine merge upstream

[stephmilovic]

@elasticmachine merge upstream

[stephmilovic]

@elasticmachine merge upstream

[kibana-ci]

:green_heart: Build Succeeded

• Buildkite Build
• Commit: c52411997571fbe9756e2b011062a365e86e0a0e

Metrics [docs]

Module Count

Fewer modules leads to a faster build time

id	before	after	diff
securitySolution	5753	5751	-2

Public APIs missing comments

Total count of every public API that lacks a comment. Target amount is 0. Run node scripts/build_api_docs --plugin [yourplugin] --stats comments for more detailed information.

id	before	after	diff
@kbn/elastic-assistant	130	132	+2

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id	before	after	diff
securitySolution	20.3MB	20.3MB	-4.3KB

Public APIs missing exports

Total count of every type that is part of your API that should be exported but is not. This will cause broken links in the API documentation system. Target amount is 0. Run node scripts/build_api_docs --plugin [yourplugin] --stats exports for more detailed information.

id	before	after	diff
@kbn/elastic-assistant	9	10	+1

Unknown metric groups

API count

id	before	after	diff
@kbn/elastic-assistant	156	158	+2

History

• :yellow_heart: Build #233274 was flaky 26b69d2fe0bde73b21d3499f7f0887b919d7859f
• :yellow_heart: Build #233137 was flaky 659619be364a4b42f3cc078691691c6044e8efbf
• :yellow_heart: Build #232889 was flaky d84530b1c9e7aae1eb8b74c77fda7b4516f94cca
• :broken_heart: Build #232868 failed 28ecc6f4462fa05795f5788bb5d8729af8450907
• :yellow_heart: Build #232554 was flaky 00deb35f4f558f6879a2639c607895d5349ce482
• :broken_heart: Build #232524 failed 555156ae18e881df1e3feccd59fd7f758830d1de

To update your PR or re-run it, just comment with: @elasticmachine merge upstream