holmesgpt
holmesgpt copied to clipboard
robusta-dev
ROB-1356: holmes is not following DNS troubleshooting instructions
Summary by CodeRabbit
-
New Features
- Introduced a secure Bash toolset with support for running validated bash and Kubernetes (kubectl) commands, including advanced argument parsing, command validation, and output handling.
- Added detailed documentation and instructions for using Bash and kubectl commands, including Kubernetes-specific troubleshooting workflows.
- Enabled configuration of allowed container images and permitted commands for kubectl executions.
-
Bug Fixes
- Improved environment variable substitution to support multiple replacements within strings and enhanced error handling for missing variables.
- Fixed test evaluation logic to better handle different correctness evaluation types and expected scores.
-
Documentation
- Added comprehensive DNS troubleshooting guides and Kubernetes manifest examples for diagnosing DNS issues in clusters.
- Included full-text Kubernetes documentation as test fixtures for reference and validation.
-
Tests
- Added extensive tests for safe command parsing, rejection of unsafe shell commands, and kubectl command round-tripping.
- Introduced new test cases and fixtures for Kubernetes DNS troubleshooting scenarios, including manifests, expected outputs, and toolset configurations.
-
Chores
- Updated
.gitignorepatterns and improved test configuration file management.
- Updated
Summary by CodeRabbit
-
New Features
- Introduced a secure Bash toolset with support for running validated bash and Kubernetes (kubectl) commands, including advanced argument parsing, command validation, and output handling.
- Added detailed documentation and instructions for using Bash and kubectl commands, including Kubernetes-specific troubleshooting workflows.
- Enabled configuration of allowed container images and permitted commands for kubectl executions.
-
Bug Fixes
- Improved environment variable substitution to support multiple replacements within strings and enhanced error handling for missing variables.
- Fixed test evaluation logic to better handle different correctness evaluation types and expected scores.
-
Documentation
- Added comprehensive DNS troubleshooting guides and Kubernetes manifest examples for diagnosing DNS issues in clusters.
- Included full-text Kubernetes documentation as test fixtures for reference and validation.
-
Tests
- Added extensive tests for safe command parsing, rejection of unsafe shell commands, and kubectl command round-tripping.
- Introduced new test cases and fixtures for Kubernetes DNS troubleshooting scenarios, including manifests, expected outputs, and toolset configurations.
-
Chores
- Updated
.gitignorepatterns and improved test configuration file management.
- Updated
Walkthrough
This update introduces a comprehensive Bash and Kubernetes command execution toolset for Holmes, including secure parsing, validation, and execution of bash and kubectl commands. It adds extensive configuration, validation utilities, and integration tests, while updating test infrastructure and prompt handling to support file inclusion, flexible evaluation, and robust environment variable replacement.
Changes
| File(s) | Change Summary |
|---|---|
.gitignore, tests/llm/fixtures/.gitignore |
Removed .AUTOGENERATED ignore line; added ignore patterns for autogenerated files in test fixtures. |
holmes/core/prompt.py, holmes/main.py |
Added append_file_to_user_prompt function; refactored prompt file inclusion to use this utility. |
holmes/plugins/toolsets/__init__.py |
Registered new BashExecutorToolset in toolsets loader. |
holmes/plugins/toolsets/bash/bash_toolset.py, holmes/plugins/toolsets/bash/bash_instructions.jinja2 |
Added Bash toolset implementation with kubectl and bash command tools, LLM instructions, and toolset registration. |
holmes/plugins/toolsets/bash/common/config.py |
Added Pydantic models for Bash executor and kubectl image/command whitelist configuration. |
holmes/plugins/toolsets/bash/common/stringify.py |
Added shell argument escaping utility function. |
holmes/plugins/toolsets/bash/common/validators.py |
Added regex and whitelist validators for argparse arguments. |
holmes/plugins/toolsets/bash/grep/__init__.py |
Added grep command parser, validator, and stringifier. |
holmes/plugins/toolsets/bash/kubectl/constants.py, kubectl_get.py, kubectl_describe.py, kubectl_events.py, kubectl_logs.py, kubectl_run.py, kubectl_top.py, __init__.py |
Added kubectl command subparsers, validators, constants, and stringifiers for get, describe, events, logs, run, and top actions. |
holmes/plugins/toolsets/bash/parse_command.py |
Added parser and safe stringifier for bash/kubectl/grep commands, with command splitting and validation logic. |
holmes/plugins/toolsets/bash/common/bash.py |
Added Bash command execution utility with structured result and error handling. |
tests/plugins/toolsets/bash/test_safe_commands.py |
Added integration tests for kubectl command parsing/stringifying round-trip. |
tests/plugins/toolsets/bash/test_incorrect_commands.py |
Added tests to ensure unsafe or unauthorized commands are rejected by the safe command parser. |
holmes/config_utils.py |
Enhanced environment variable replacement to support multiple variables per string and full-string substitution. |
tests/test_load_config.py |
Added unit tests for environment variable replacement logic, covering success and error cases. |
tests/llm/conftest.py |
Improved correctness score extraction for test summary reporting. |
tests/llm/test_ask_holmes.py, tests/llm/test_investigate.py |
Updated test logic to support new Evaluation class, flexible evaluation types, and improved correctness assertion. |
tests/llm/utils/braintrust.py |
Added metadata field to evaluation logging. |
tests/llm/utils/classifiers.py |
Added "loose" evaluation mode for correctness classifier with updated prompt and logic. |
tests/llm/utils/mock_utils.py |
Refactored evaluation models; added support for including files in prompts; updated test case loading logic. |
tests/llm/fixtures/** |
Added or updated numerous test fixtures, manifest files, test cases, toolset configs, and documentation for DNS/Kubernetes troubleshooting scenarios. |
Sequence Diagram(s)
sequenceDiagram
participant User
participant HolmesCLI
participant PromptUtil
participant BashToolset
participant BashCommandValidator
participant BashExecutor
User->>HolmesCLI: ask --file file1 --file file2
HolmesCLI->>PromptUtil: append_file_to_user_prompt(prompt, file1)
PromptUtil-->>HolmesCLI: prompt + file1 content
HolmesCLI->>PromptUtil: append_file_to_user_prompt(prompt, file2)
PromptUtil-->>HolmesCLI: prompt + file2 content
HolmesCLI->>BashToolset: execute(command)
BashToolset->>BashCommandValidator: validate(command)
BashCommandValidator-->>BashToolset: validated command
BashToolset->>BashExecutor: execute_bash_command(validated command)
BashExecutor-->>BashToolset: StructuredToolResult
BashToolset-->>HolmesCLI: result
HolmesCLI-->>User: result
sequenceDiagram
participant TestRunner
participant TestCaseLoader
participant PromptUtil
participant LLMClassifier
TestRunner->>TestCaseLoader: load_test_cases()
TestCaseLoader->>PromptUtil: load_include_files(folder, include_files)
PromptUtil-->>TestCaseLoader: appended prompt
TestCaseLoader-->>TestRunner: test cases with prompts
TestRunner->>LLMClassifier: evaluate_correctness(expected, output, evaluation_type)
LLMClassifier-->>TestRunner: score
TestRunner-->>TestRunner: assert score vs expected
Suggested labels
enhancement
Suggested reviewers
- moshemorad
- arikalon1
📜 Recent review details
Configuration used: CodeRabbit UI Review profile: CHILL Plan: Pro
📥 Commits
Reviewing files that changed from the base of the PR and between 61597ec1e65231f6223225403d930a07f6f21e8b and 10474c26a4eefa615dc69b19d151687240748d16.
📒 Files selected for processing (1)
holmes/plugins/toolsets/bash/bash_toolset.py(1 hunks)
🚧 Files skipped from review as they are similar to previous changes (1)
- holmes/plugins/toolsets/bash/bash_toolset.py
⏰ Context from checks skipped due to timeout of 90000ms (4)
- GitHub Check: build (3.11)
- GitHub Check: build (3.12)
- GitHub Check: build (3.10)
- GitHub Check: build (3.12)
✨ Finishing Touches
- [ ] 📝 Generate Docstrings
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.
🪧 Tips
Chat
There are 3 ways to chat with CodeRabbit:
- Review comments: Directly reply to a review comment made by CodeRabbit. Example:
I pushed a fix in commit <commit_id>, please review it.Explain this complex logic.Open a follow-up GitHub issue for this discussion.
- Files and specific lines of code (under the "Files changed" tab): Tag
@coderabbitaiin a new review comment at the desired location with your query. Examples:@coderabbitai explain this code block.@coderabbitai modularize this function.
- PR comments: Tag
@coderabbitaiin a new PR comment to ask questions about the PR branch. For the best results, please provide a very specific query, as very limited context is provided in this mode. Examples:@coderabbitai gather interesting stats about this repository and render them as a table. Additionally, render a pie chart showing the language distribution in the codebase.@coderabbitai read src/utils.ts and explain its main purpose.@coderabbitai read the files in the src/scheduler package and generate a class diagram using mermaid and a README in the markdown format.@coderabbitai help me debug CodeRabbit configuration file.
Support
Need help? Create a ticket on our support page for assistance with any issues or questions.
Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments.
CodeRabbit Commands (Invoked using PR comments)
@coderabbitai pauseto pause the reviews on a PR.@coderabbitai resumeto resume the paused reviews.@coderabbitai reviewto trigger an incremental review. This is useful when automatic reviews are disabled for the repository.@coderabbitai full reviewto do a full review from scratch and review all the files again.@coderabbitai summaryto regenerate the summary of the PR.@coderabbitai generate docstringsto generate docstrings for this PR.@coderabbitai generate sequence diagramto generate a sequence diagram of the changes in this PR.@coderabbitai resolveresolve all the CodeRabbit review comments.@coderabbitai configurationto show the current CodeRabbit configuration for the repository.@coderabbitai helpto get help.
Other keywords and placeholders
- Add
@coderabbitai ignoreanywhere in the PR description to prevent this PR from being reviewed. - Add
@coderabbitai summaryto generate the high-level summary at a specific location in the PR description. - Add
@coderabbitaianywhere in the PR title to generate the title automatically.
CodeRabbit Configuration File (.coderabbit.yaml)
- You can programmatically configure CodeRabbit by adding a
.coderabbit.yamlfile to the root of your repository. - Please see the configuration documentation for more information.
- If your editor has YAML language server enabled, you can add the path at the top of this file to enable auto-completion and validation:
# yaml-language-server: $schema=https://coderabbit.ai/integrations/schema.v2.json
Documentation and Community
- Visit our Documentation for detailed information on how to use CodeRabbit.
- Join our Discord Community to get help, request features, and share feedback.
- Follow us on X/Twitter for updates and announcements.
![coderabbitai[bot] avatar](https://avatars.githubusercontent.com/in/347564?v=4 "Published by a pub.dev verified publisher"){kind=small}
Results of HolmesGPT evals
- ask_holmes: 42/53 test cases were successful
- investigate: 15/15 test cases were successful
Legend
- :white_check_mark: the test was successful
- :warning: the test failed but is known to be flakky or known to fail
- :x: the test failed and should be fixed before merging the PR
![github-actions[bot] avatar](https://avatars.githubusercontent.com/in/15368?v=4 "Published by a pub.dev verified publisher"){kind=small}