gateway icon indicating copy to clipboard operation
gateway copied to clipboard

Feat/record and play framework

Open WuonParticle opened this issue 5 months ago • 2 comments

Before submitting this PR, please make sure:

  • [x] Your code builds clean without any errors or warnings
  • [x] You are using approved title ("feat/", "fix/", "docs/", "refactor/")

A description of the changes proposed in the pull request:

  • New Features

    • Introduced a comprehensive "Record and Play" (RnP) testing framework with documentation, a reusable test harness, and dependency mocking strategies.
    • Added modular Fastify API endpoints and a sample API demonstrating diverse dependency scenarios.
    • Implemented snapshot-based API test cases with mock data management ensuring deterministic unit tests.
    • Provided custom serialization support for complex data types like BigNumber in tests.
    • Added VS Code configuration guide for managing multiple Jest test suites and debugging workflows.
  • Bug Fixes

    • Updated test cases to expect correct HTTP status codes (400 instead of 500) for invalid input scenarios.
  • Chores

    • Enhanced test scripts, configuration, and environment variable handling for improved test isolation and reproducibility.
    • Added new test setup and configuration files to support the RnP framework and snapshot testing.
  • Documentation

    • Added a detailed guide explaining the RnP testing workflow, conventions, and best practices.

Tests performed by the developer: pnpm test-record (changes files intentionally) pnpm test rnp

Tips for QA testing:

  1. run pnpm test - verify that all tests pass
  2. run pnpm test rnp - verify that only tests with rnp in the name are run and pass
  3. run pnpm test-play - verify that only play tests are run and pass
  4. run pnpm test-record - verify that rnpExample mock files and snapshot file changes change
  5. run pnpm test-play - verify that only rnp tests are run and pass
  6. revert changes to rnpExample.test.ts.snap
  7. run pnpm test-play - verify that rnpExample play tests relying on mocks fail
  8. revert all changes to mock files
  9. run pnpm test-play - verify that tests pass

WuonParticle avatar Jul 17 '25 21:07 WuonParticle

Separated the "Play" tests from the primary test suite today with https://github.com/hummingbot/gateway/pull/490/commits/9bd329a5e0bf7810321a232fcae0a7d3cb721ed2 as LLM agents were running the unit test suite with -u to force snapshots to avoid having to actually debug issues. Also clarified everywhere "Record" and "Play" Tests rather than saying unit.

WuonParticle avatar Jul 25 '25 00:07 WuonParticle

I recommend doing a squash merge when eventually merging this as 42 commits is pretty outrageous for the functionality being added. However, I think keeping the git history in the PR is important for posterity and to understand why some design decisions were made.

WuonParticle avatar Aug 15 '25 19:08 WuonParticle