PyAirbyte icon indicating copy to clipboard operation
PyAirbyte copied to clipboard

feat: add cloud cleanup script for stale integration test artifacts

Open aaronsteers opened this issue 2 months ago • 5 comments

This PR targets the following PR:

  • #782

feat: add cloud cleanup script for stale integration test artifacts

Summary

Adds a helper script bin/cleanup_cloud_artifacts.py to clean up stale connector and connection definitions left behind by cloud integration tests. The script:

  • 🔍 Scans source code for hardcoded UUIDs using regex and exempts them from cleanup
  • 📋 Lists connections, sources, and destinations using existing PyAirbyte cloud operations
  • 🛡️ Defaults to --dry-run mode for safety, requires --no-dry-run for actual deletions
  • 🔗 Prints web URLs for resources that would be deleted in dry-run mode
  • ⚙️ Uses existing authentication patterns via environment variables
  • 🚀 Includes poe cloud-cleanup task for easy execution

Review & Testing Checklist for Human (4 items)

  • [ ] Test end-to-end with real cloud credentials - This is critical since I couldn't test actual cloud operations due to missing credentials
  • [ ] Verify UUID exemption logic works correctly - Test that hardcoded UUIDs in the codebase are actually skipped during deletion
  • [ ] Test both dry-run and actual deletion modes - Ensure dry-run doesn't accidentally delete anything and that --no-dry-run works as expected
  • [ ] Verify error handling with invalid credentials/network issues - Test graceful failure scenarios

Test Plan

  1. Run poe cloud-cleanup (dry-run mode) with valid cloud credentials to see what would be deleted
  2. Verify that known hardcoded UUIDs from test files are listed as exempted
  3. Create a test connector/connection and verify it appears in the deletion list
  4. Test poe cloud-cleanup --no-dry-run on the test resources to verify actual deletion works

Notes

  • Script successfully scans source code and found 11 hardcoded UUIDs during local testing
  • All linting, formatting, and type checking passes
  • CLI interface works correctly with --help, --dry-run, and --no-dry-run flags
  • Could not test actual cloud workspace operations due to missing credentials in dev environment

Session: https://app.devin.ai/sessions/b580c93f6f144ca8b141dce686201892
Requested by: @aaronsteers

aaronsteers avatar Sep 19 '25 21:09 aaronsteers