PyAirbyte icon indicating copy to clipboard operation
PyAirbyte copied to clipboard

feat: Add Hercules AI test agent example (do not merge)

Open aaronsteers opened this issue 3 weeks ago • 6 comments

feat: Add Hercules AI test agent example (do not merge)

Summary

This PR adds an example demonstrating how to use Hercules (an open-source AI testing agent) to test PyAirbyte functionality. This is the first part of a "bake off" comparison between Hercules and Goose AI test agents.

Key additions:

  • Documentation on using Hercules with PyAirbyte in examples/ai-test-agents/hercules/README.md
  • Gherkin feature file with two test scenarios (test_pyairbyte.feature)
  • Requirements file for Hercules dependencies (requirements.txt)

Important note: Due to a dependency conflict (Hercules requires psutil <6.0.0 while airbyte-cdk requires psutil 6.1.0), Hercules must be installed in a separate virtual environment. This example documents the workaround approach rather than integrating Hercules as a direct dependency.

Review & Testing Checklist for Human

This is a YELLOW risk PR (documentation/example only, but untested):

  • [ ] Verify the Gherkin test scenarios are valid - I created these based on PyAirbyte's API but have NOT tested them with Hercules
  • [ ] Check if the separate environment approach aligns with the "bake off" intent - This is more of a "how to use together" guide than a true integration
  • [ ] Review the installation instructions - Confirm the Hercules installation steps are correct and complete

Test Plan

To verify this example works:

  1. Create a separate virtual environment
  2. Install Hercules following the README instructions
  3. Set up an OpenAI API key
  4. Run Hercules with the provided feature file
  5. Verify the test scenarios execute successfully

Notes

  • This is part 1 of 2 for the AI test agent bake-off (Goose implementation still pending)
  • The dependency conflict prevents Hercules from being a direct PyAirbyte dependency
  • The Gherkin scenarios are untested examples that may need refinement
  • Requested by: AJ Steers ([email protected], @aaronsteers)
  • Devin session: https://app.devin.ai/sessions/0d255eb4e8c14c6a93cd7cbc5c8f9a72

Summary by CodeRabbit

  • Documentation

    • Added comprehensive guide for using Hercules AI testing agent to test PyAirbyte, including setup and usage instructions.
  • Tests

    • Added test scenarios for PyAirbyte integration, covering basic functionality and connector discovery workflows.

aaronsteers avatar Nov 12 '25 05:11 aaronsteers