[Security Solution] AI Assistant - System prompt move

[stephmilovic]

[!CAUTION] Opening PR to get CI green, but not ready for review as I have not completed prompt eval on this work.

Summary

This PR refactors the system prompt strategy to streamline the assistant's performance and reduce issues arising from outdated system prompts. sers will have a more focused AI Assistant, with the flexibility to add custom prompts as needed.

Key Changes

1. Default System Prompt Removal from Client:

   • Removed the default system prompt from the AI Assistant.
     • Removed default system prompts from the creation flow in security_solution/public/assistant/provider.tsx
     • Excluded any previously created default system prompts by adding filter to the query in elastic_assistant/server/routes/prompts/find_route.ts

2. Default System Prompt Elements Migrated to LangGraph:

   • Key elements of the previous system prompts have been integrated directly into the LangGraph, ensuring that the assistant remains focused on relevant topics without the need for a separate system prompt.
   • Reworked the assistant’s focus to ensure it only responds to inquiries related to security, Elastic security, and security operations. This jailing mechanism is now part of the agent prompts in LangGraph.

3. Moved Custom (User) System Prompt Sending Logic to LangGraph

   • Custom system prompts still exist in the UI. They are no longer sent in the first user message, but instead appended to the default system prompt located in the LangGraph.
   • Once the first message of the conversation has been sent, the system prompt is prepended as a message in the conversation, where the user is "System"
   • Side effect: Since the first user message no longer contains the system prompt, if the user changes the system prompt mid conversation in the assistant settings, that change will take effect immediately without clearing the conversation. For example (see trace):

4. Clearing the system prompt now updates the conversation

   • Previously, when a custom system prompt was cleared in the EmptyConvo UI, it was only cleared from the message about to be sent, not from the conversation. You could "unclear" it to bring it back:

     https://github.com/user-attachments/assets/62d5305f-781b-4505-af10-e60bfe06352c

   • Now, when a custom system prompt was cleared in the EmptyConvo UI, an API call is sent to remove the system prompt from the conversation. This is because we are adding the custom system prompt to the user message, and now we are adding it to the LangGraph system prompt on the server side. So the conversation apiConfig.defaultSystemPromptId needs to stay up to date with the UI so we know whether or not to send the prompt.

     https://github.com/user-attachments/assets/fc3c669e-7d76-4f59-9526-909af2b1dfbc

5. General System Prompt Flow Cleanup

   • Resolved an issue where the conversation would change unexpectedly after hitting "Save" when modifying the system prompts' default conversations. The root cause was that AssistantSettings was expecting the conversationSettings object to be indexed by id, while in some cases, it was indexed by title as a fallback. Additionally, there was an instance in SystemPromptEditor where the code called setConversationSettings and incorrectly keyed the update by title instead of id. To fix this, I made two changes. In AssistantSettings, I modified the code that referenced conversationSettings[cId] to use !Object.values(conversationSettings).some(({ id }) => id === cId) to verify the existence of the conversation. Secondly, in SystemPromptEditor, I updated the code to ensure the object is consistently indexed by id instead of title. These adjustments have resolved the bug, ensuring that conversations remain consistent after saving changes. This is what that bug looked like:

     https://github.com/user-attachments/assets/2043117b-b01a-4c81-b3ae-f08e4d6a496d

   • Resolved an issue preventing a conversation's apiConfig.defaultSystemPromptId from being set to undefined after it had been assigned a value. The getUpdateScript function used by the bulk actions route only targeted updates where default_system_prompt_id had a defined value, overlooking cases where it was undefined. By adding a condition to remove the property when it is undefined, this bug was fixed. This is what that bug looked like:

     https://github.com/user-attachments/assets/a0048e46-a09c-4091-a611-c67f7ef6c8fa