PyAirbyte
PyAirbyte copied to clipboard
feat: Add Hercules AI test agent example (do not merge)
feat: Add Hercules AI test agent example (do not merge)
Summary
This PR adds an example demonstrating how to use Hercules (an open-source AI testing agent) to test PyAirbyte functionality. This is the first part of a "bake off" comparison between Hercules and Goose AI test agents.
Key additions:
- Documentation on using Hercules with PyAirbyte in
examples/ai-test-agents/hercules/README.md - Gherkin feature file with two test scenarios (
test_pyairbyte.feature) - Requirements file for Hercules dependencies (
requirements.txt)
Important note: Due to a dependency conflict (Hercules requires psutil <6.0.0 while airbyte-cdk requires psutil 6.1.0), Hercules must be installed in a separate virtual environment. This example documents the workaround approach rather than integrating Hercules as a direct dependency.
Review & Testing Checklist for Human
This is a YELLOW risk PR (documentation/example only, but untested):
- [ ] Verify the Gherkin test scenarios are valid - I created these based on PyAirbyte's API but have NOT tested them with Hercules
- [ ] Check if the separate environment approach aligns with the "bake off" intent - This is more of a "how to use together" guide than a true integration
- [ ] Review the installation instructions - Confirm the Hercules installation steps are correct and complete
Test Plan
To verify this example works:
- Create a separate virtual environment
- Install Hercules following the README instructions
- Set up an OpenAI API key
- Run Hercules with the provided feature file
- Verify the test scenarios execute successfully
Notes
- This is part 1 of 2 for the AI test agent bake-off (Goose implementation still pending)
- The dependency conflict prevents Hercules from being a direct PyAirbyte dependency
- The Gherkin scenarios are untested examples that may need refinement
- Requested by: AJ Steers ([email protected], @aaronsteers)
- Devin session: https://app.devin.ai/sessions/0d255eb4e8c14c6a93cd7cbc5c8f9a72
Summary by CodeRabbit
-
Documentation
- Added comprehensive guide for using Hercules AI testing agent to test PyAirbyte, including setup and usage instructions.
-
Tests
- Added test scenarios for PyAirbyte integration, covering basic functionality and connector discovery workflows.