holmesgpt
holmesgpt copied to clipboard
robusta-dev
ROB-1347: add more evals for the new logging API
[!CAUTION]
Review failed
The pull request is closed.
Walkthrough
The changes introduce a refactored approach to Kubernetes log fetching and processing, with structured log parsing, advanced filtering, and improved formatting. New utilities and tests support time conversion and log handling. Test infrastructure is enhanced to support conversation history and more detailed tool call tracking. Comprehensive fixtures and Helm manifests are added for robust integration and unit testing.
Changes
| File(s) | Change Summary |
|---|---|
| holmes/plugins/toolsets/kubernetes_logs.py | Refactored log fetching to use a new StructuredLog data model, added advanced log parsing, filtering, and formatting; improved error handling and diagnostics. |
| holmes/plugins/toolsets/utils.py | Added to_unix_ms function for converting timestamp strings to Unix milliseconds. |
| tests/plugins/toolsets/kubernetes/test_kubernetes_logs_mocked.py | Centralized and streamlined Kubernetes log mocking via mock_k8s_read_log; updated tests to use this mock and adjusted assertions for new log outputs. |
| tests/plugins/toolsets/kubernetes/test_kubernetes_logs_unit.py | Added new unit tests for log parsing and filtering, covering various scenarios and edge cases. |
| tests/plugins/toolsets/test_toolset_utils.py | Refactored timestamp processing test; added tests for to_unix_ms. |
| tests/llm/test_ask_holmes.py | Enhanced tool call output to include parameters; now passes conversation history to chat message builder. |
| tests/llm/utils/mock_utils.py | Added support for loading conversation history in test cases; introduced conversation_history field and load_conversation_history function. |
| tests/llm/fixtures/test_ask_holmes/42_slack_deployment_logs/conversation_history/... | Added system and assistant markdown files for AI troubleshooting prompts and alert summaries. |
| tests/llm/fixtures/test_ask_holmes/42_slack_deployment_logs/helm/sock-shop.yaml | Added comprehensive Helm manifest for the "sock-shop" microservices demo application. |
| tests/llm/fixtures/test_ask_holmes/42_slack_deployment_logs/*.txt, *.yaml | Added various test fixture files simulating Kubernetes tool operations, log fetching, and alerting scenarios. |
| tests/llm/fixtures/test_ask_holmes/43_slack_statefulset_logs/conversation_history/... | Added system prompt and assistant markdown for statefulset-related troubleshooting. |
| tests/llm/fixtures/test_ask_holmes/43_slack_statefulset_logs/helm/sock-shop.yaml | Added Helm manifest for "sock-shop" in statefulset test scenario. |
| tests/llm/fixtures/test_ask_holmes/43_slack_statefulset_logs/test_case.yaml | Added test case YAML for statefulset log retrieval scenario. |
Sequence Diagram(s)
sequenceDiagram
participant User
participant Toolset
participant KubernetesAPI
User->>Toolset: fetch_pod_logs(params)
Toolset->>KubernetesAPI: fetch logs (current, timestamps)
alt multiple containers
Toolset->>KubernetesAPI: fetch logs for each container
end
Toolset->>KubernetesAPI: fetch previous logs (if needed)
Toolset->>Toolset: parse_logs(raw_logs)
Toolset->>Toolset: filter_logs(structured_logs, params)
Toolset->>Toolset: format_logs(filtered_logs, display_container_name)
Toolset-->>User: StructuredToolResult (logs or error)
Suggested labels
enhancement
Suggested reviewers
- arikalon1
📜 Recent review details
Configuration used: CodeRabbit UI Review profile: CHILL Plan: Pro
📥 Commits
Reviewing files that changed from the base of the PR and between b3986acbd93a2806b140ee4f444f1e78ebb66b4e and a664fb7dc15c4dffb6102ab587e8b61a5b6e6b3b.
📒 Files selected for processing (23)
holmes/plugins/toolsets/kubernetes_logs.py(7 hunks)holmes/plugins/toolsets/utils.py(1 hunks)tests/llm/fixtures/test_ask_holmes/42_slack_deployment_logs/conversation_history/00_system.md(1 hunks)tests/llm/fixtures/test_ask_holmes/42_slack_deployment_logs/conversation_history/01_assistant.md(1 hunks)tests/llm/fixtures/test_ask_holmes/42_slack_deployment_logs/conversation_history/02_assistant.md(1 hunks)tests/llm/fixtures/test_ask_holmes/42_slack_deployment_logs/conversation_history/03_assistant.md(1 hunks)tests/llm/fixtures/test_ask_holmes/42_slack_deployment_logs/fetch_pod_logs.txt(1 hunks)tests/llm/fixtures/test_ask_holmes/42_slack_deployment_logs/get_current_time.txt(1 hunks)tests/llm/fixtures/test_ask_holmes/42_slack_deployment_logs/helm/sock-shop.yaml(1 hunks)tests/llm/fixtures/test_ask_holmes/42_slack_deployment_logs/kubectl_find_resource.txt(1 hunks)tests/llm/fixtures/test_ask_holmes/42_slack_deployment_logs/kubectl_get_by_kind_in_namespace.txt(1 hunks)tests/llm/fixtures/test_ask_holmes/42_slack_deployment_logs/kubectl_get_by_name.txt(1 hunks)tests/llm/fixtures/test_ask_holmes/42_slack_deployment_logs/kubectl_lineage_children.txt(1 hunks)tests/llm/fixtures/test_ask_holmes/42_slack_deployment_logs/test_case.yaml(1 hunks)tests/llm/fixtures/test_ask_holmes/43_slack_statefulset_logs/conversation_history/00_system.md(1 hunks)tests/llm/fixtures/test_ask_holmes/43_slack_statefulset_logs/conversation_history/01_assistant.md(1 hunks)tests/llm/fixtures/test_ask_holmes/43_slack_statefulset_logs/helm/sock-shop.yaml(1 hunks)tests/llm/fixtures/test_ask_holmes/43_slack_statefulset_logs/test_case.yaml(1 hunks)tests/llm/test_ask_holmes.py(2 hunks)tests/llm/utils/mock_utils.py(3 hunks)tests/plugins/toolsets/kubernetes/test_kubernetes_logs_mocked.py(7 hunks)tests/plugins/toolsets/kubernetes/test_kubernetes_logs_unit.py(1 hunks)tests/plugins/toolsets/test_toolset_utils.py(1 hunks)
✨ Finishing Touches
- [ ] 📝 Generate Docstrings
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.
🪧 Tips
Chat
There are 3 ways to chat with CodeRabbit:
- Review comments: Directly reply to a review comment made by CodeRabbit. Example:
I pushed a fix in commit <commit_id>, please review it.Explain this complex logic.Open a follow-up GitHub issue for this discussion.
- Files and specific lines of code (under the "Files changed" tab): Tag
@coderabbitaiin a new review comment at the desired location with your query. Examples:@coderabbitai explain this code block.@coderabbitai modularize this function.
- PR comments: Tag
@coderabbitaiin a new PR comment to ask questions about the PR branch. For the best results, please provide a very specific query, as very limited context is provided in this mode. Examples:@coderabbitai gather interesting stats about this repository and render them as a table. Additionally, render a pie chart showing the language distribution in the codebase.@coderabbitai read src/utils.ts and explain its main purpose.@coderabbitai read the files in the src/scheduler package and generate a class diagram using mermaid and a README in the markdown format.@coderabbitai help me debug CodeRabbit configuration file.
Support
Need help? Create a ticket on our support page for assistance with any issues or questions.
Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments.
CodeRabbit Commands (Invoked using PR comments)
@coderabbitai pauseto pause the reviews on a PR.@coderabbitai resumeto resume the paused reviews.@coderabbitai reviewto trigger an incremental review. This is useful when automatic reviews are disabled for the repository.@coderabbitai full reviewto do a full review from scratch and review all the files again.@coderabbitai summaryto regenerate the summary of the PR.@coderabbitai generate docstringsto generate docstrings for this PR.@coderabbitai generate sequence diagramto generate a sequence diagram of the changes in this PR.@coderabbitai resolveresolve all the CodeRabbit review comments.@coderabbitai configurationto show the current CodeRabbit configuration for the repository.@coderabbitai helpto get help.
Other keywords and placeholders
- Add
@coderabbitai ignoreanywhere in the PR description to prevent this PR from being reviewed. - Add
@coderabbitai summaryto generate the high-level summary at a specific location in the PR description. - Add
@coderabbitaianywhere in the PR title to generate the title automatically.
CodeRabbit Configuration File (.coderabbit.yaml)
- You can programmatically configure CodeRabbit by adding a
.coderabbit.yamlfile to the root of your repository. - Please see the configuration documentation for more information.
- If your editor has YAML language server enabled, you can add the path at the top of this file to enable auto-completion and validation:
# yaml-language-server: $schema=https://coderabbit.ai/integrations/schema.v2.json
Documentation and Community
- Visit our Documentation for detailed information on how to use CodeRabbit.
- Join our Discord Community to get help, request features, and share feedback.
- Follow us on X/Twitter for updates and announcements.
![coderabbitai[bot] avatar](https://avatars.githubusercontent.com/in/347564?v=4 "Published by a pub.dev verified publisher"){kind=small}
Results of HolmesGPT evals
- ask_holmes: 35/43 test cases were successful
- investigate: 13/15 test cases were successful
Legend
- :white_check_mark: the test was successful
- :warning: the test failed but is known to be flakky or known to fail
- :x: the test failed and should be fixed before merging the PR
![github-actions[bot] avatar](https://avatars.githubusercontent.com/in/15368?v=4 "Published by a pub.dev verified publisher"){kind=small}