holmesgpt
holmesgpt copied to clipboard
robusta-dev
Rob 1254 holmes send toolset version json schema
Summary by CodeRabbit
- New Features
- Toolsets now include version information, default status, and configuration schema details where available.
- Added a feature flag to enable Holmes toolsets from SaaS environments.
- Bug Fixes
- Improved synchronization to ensure new metadata fields are correctly handled.
- Tests
- Enhanced test coverage to verify new toolset metadata fields and configuration schema handling.
Summary by CodeRabbit
- New Features
- Toolsets now include version information, default status, and configuration schema details where available.
- Added a feature flag to enable Holmes toolsets from SaaS environments.
- Bug Fixes
- Improved synchronization to ensure new metadata fields are correctly handled.
- Tests
- Enhanced test coverage to verify new toolset metadata fields and configuration schema handling.
Summary by CodeRabbit
- New Features
- Toolsets now include version information, default status, and configuration schema details where available.
- Added a feature flag to enable Holmes toolsets from SaaS environments.
- Bug Fixes
- Improved synchronization to ensure new metadata fields are correctly handled.
- Tests
- Enhanced test coverage to verify new toolset metadata fields and configuration schema handling.
Walkthrough
The changes introduce new metadata and schema-related attributes to toolset classes and their database models, including configuration schema and version information. Synchronization logic is updated to handle these attributes, and tests are expanded to validate the new fields and their correct handling in both basic and configuration-aware toolsets.
Changes
| File(s) | Change Summary |
|---|---|
| holmes/core/tools.py | Added config_class and version as class variables to Toolset; added property is_configured_locally; extended ToolsetDBModel with version, config_schema, is_default, and is_configured_locally fields. |
| holmes/plugins/toolsets/grafana/base_grafana_toolset.py | Removed type annotations for config_class in BaseGrafanaToolset, leaving it as a plain assignment. |
| holmes/plugins/toolsets/opensearch/opensearch.py | Added config_class = OpenSearchConfig and version = "0.0.1" to OpenSearchToolset. |
| holmes/plugins/toolsets/opensearch/opensearch_utils.py | Added config_class = OpenSearchIndexConfig to BaseOpenSearchToolset. |
| holmes/utils/holmes_sync_toolsets.py | Updated holmes_sync_toolsets_status to include config_schema and version in toolset synchronization, generating schema from config_class if present; conditionally exclude these fields based on environment variable. |
| tests/test_holmes_sync_toolsets.py | Added SampleConfig class; enhanced tests to check for is_default, config_schema, version, and is_configured_locally; added tests for toolsets with configuration schema and local configuration detection. |
| holmes/common/env_vars.py | Added new boolean environment variable flag ENABLE_HOLMES_TOOLSETS_FROM_SAAS initialized to False. |
| holmes/plugins/toolsets/coralogix/toolset_coralogix_logs.py | Added config_class = CoralogixConfig and version = "0.0.1" to BaseCoralogixToolset; added version = "0.0.1" to CoralogixLogsToolset. |
| holmes/plugins/toolsets/datadog.py | Added config_class = DatadogConfig and version = "0.0.1" to DatadogToolset. |
| holmes/plugins/toolsets/grafana/toolset_grafana.py | Added version = "0.0.1" to GrafanaToolset. |
| holmes/plugins/toolsets/grafana/toolset_grafana_loki.py | Added version = "0.0.1" to BaseGrafanaLokiToolset. |
| holmes/plugins/toolsets/grafana/toolset_grafana_tempo.py | Added version = "0.0.1" to BaseGrafanaTempoToolset. |
| holmes/plugins/toolsets/internet/internet.py | Added version = "0.0.1" to InternetBaseToolset. |
| holmes/plugins/toolsets/kafka.py | Added config_class = KafkaConfig and version = "0.0.1" to KafkaToolset. |
| holmes/plugins/toolsets/newrelic.py | Added config_class = NewrelicConfig and version = "0.0.1" to NewRelicToolset. |
| holmes/plugins/toolsets/opensearch/opensearch_logs.py | Added version = "0.0.1" to OpenSearchLogsToolset. |
| holmes/plugins/toolsets/opensearch/opensearch_traces.py | Added version = "0.0.1" to OpenSearchTracesToolset. |
| holmes/plugins/toolsets/prometheus/prometheus.py | Added config_class = PrometheusConfig and version = "0.0.1" to PrometheusToolset. |
| holmes/plugins/toolsets/rabbitmq/toolset_rabbitmq.py | Added config_class = RabbitMQConfig and version = "0.0.1" to RabbitMQToolset. |
| holmes/plugins/toolsets/robusta/robusta.py | Added version = "0.0.1" to RobustaToolset. |
| Makefile | Added new target test-not-llm to run pytest excluding tests marked with llm. |
| holmes/plugins/toolsets/grafana/common.py | Updated GrafanaConfig.url attribute to use Field(...) with a description. |
Sequence Diagram(s)
sequenceDiagram
participant SyncScript as holmes_sync_toolsets_status
participant Toolset as Toolset
participant DBModel as ToolsetDBModel
SyncScript->>Toolset: Inspect config_class and version
alt Toolset has config_class
SyncScript->>Toolset: Generate config_schema from config_class
else Toolset has no config_class
SyncScript->>Toolset: Set config_schema to None
end
SyncScript->>DBModel: Create ToolsetDBModel with config_schema, version, is_default, is_configured_locally
DBModel-->>SyncScript: ToolsetDBModel instance ready for sync
Possibly related PRs
- robusta-dev/holmesgpt#459: Related core toolset metadata enhancements and refactoring of toolset loading and management.
- robusta-dev/holmesgpt#444: Introduces a BashExecutorToolset plugin that may benefit from the new toolset metadata fields added here.
Suggested labels
enhancement
Suggested reviewers
- arikalon1
📜 Recent review details
Configuration used: CodeRabbit UI Review profile: CHILL Plan: Pro
📥 Commits
Reviewing files that changed from the base of the PR and between 55393f3a56b8a5216586793b24c2520efe5dd04a and 1e6c951f06941acdbe174a8c3fa42206f01931f6.
📒 Files selected for processing (1)
holmes/plugins/toolsets/prometheus/prometheus.py(1 hunks)
🚧 Files skipped from review as they are similar to previous changes (1)
- holmes/plugins/toolsets/prometheus/prometheus.py
⏰ Context from checks skipped due to timeout of 90000ms (7)
- GitHub Check: build (3.11)
- GitHub Check: build (3.12)
- GitHub Check: build (3.10)
- GitHub Check: build (3.12)
- GitHub Check: build (3.10)
- GitHub Check: build (3.11)
- GitHub Check: build (3.12)
✨ Finishing Touches
- [ ] 📝 Generate Docstrings
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.
🪧 Tips
Chat
There are 3 ways to chat with CodeRabbit:
- Review comments: Directly reply to a review comment made by CodeRabbit. Example:
I pushed a fix in commit <commit_id>, please review it.Explain this complex logic.Open a follow-up GitHub issue for this discussion.
- Files and specific lines of code (under the "Files changed" tab): Tag
@coderabbitaiin a new review comment at the desired location with your query. Examples:@coderabbitai explain this code block.@coderabbitai modularize this function.
- PR comments: Tag
@coderabbitaiin a new PR comment to ask questions about the PR branch. For the best results, please provide a very specific query, as very limited context is provided in this mode. Examples:@coderabbitai gather interesting stats about this repository and render them as a table. Additionally, render a pie chart showing the language distribution in the codebase.@coderabbitai read src/utils.ts and explain its main purpose.@coderabbitai read the files in the src/scheduler package and generate a class diagram using mermaid and a README in the markdown format.@coderabbitai help me debug CodeRabbit configuration file.
Support
Need help? Create a ticket on our support page for assistance with any issues or questions.
Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments.
CodeRabbit Commands (Invoked using PR comments)
@coderabbitai pauseto pause the reviews on a PR.@coderabbitai resumeto resume the paused reviews.@coderabbitai reviewto trigger an incremental review. This is useful when automatic reviews are disabled for the repository.@coderabbitai full reviewto do a full review from scratch and review all the files again.@coderabbitai summaryto regenerate the summary of the PR.@coderabbitai generate docstringsto generate docstrings for this PR.@coderabbitai generate sequence diagramto generate a sequence diagram of the changes in this PR.@coderabbitai resolveresolve all the CodeRabbit review comments.@coderabbitai configurationto show the current CodeRabbit configuration for the repository.@coderabbitai helpto get help.
Other keywords and placeholders
- Add
@coderabbitai ignoreanywhere in the PR description to prevent this PR from being reviewed. - Add
@coderabbitai summaryto generate the high-level summary at a specific location in the PR description. - Add
@coderabbitaianywhere in the PR title to generate the title automatically.
CodeRabbit Configuration File (.coderabbit.yaml)
- You can programmatically configure CodeRabbit by adding a
.coderabbit.yamlfile to the root of your repository. - Please see the configuration documentation for more information.
- If your editor has YAML language server enabled, you can add the path at the top of this file to enable auto-completion and validation:
# yaml-language-server: $schema=https://coderabbit.ai/integrations/schema.v2.json
Documentation and Community
- Visit our Documentation for detailed information on how to use CodeRabbit.
- Join our Discord Community to get help, request features, and share feedback.
- Follow us on X/Twitter for updates and announcements.
![coderabbitai[bot] avatar](https://avatars.githubusercontent.com/in/347564?v=4 "Published by a pub.dev verified publisher"){kind=small}
Results of HolmesGPT evals
- ask_holmes: 42/53 test cases were successful
- investigate: 15/15 test cases were successful
Legend
- :white_check_mark: the test was successful
- :warning: the test failed but is known to be flakky or known to fail
- :x: the test failed and should be fixed before merging the PR
![github-actions[bot] avatar](https://avatars.githubusercontent.com/in/15368?v=4 "Published by a pub.dev verified publisher"){kind=small}
LGTM
Do we also support version:
- Yaml toolsets
- MCP toolsets
- Git
- Bash
In general, does every toolset needs a version?
@arikalon1 Yes, i have another task to align all toolsets properties