scaffolding icon indicating copy to clipboard operation
scaffolding copied to clipboard

chore: Introduce CLI tests to ensure regression is caught in CI

Open c12i opened this issue 1 year ago • 1 comments

Currently, the scaffolding CLI testing relies heavily on manual processes, which has led to several challenges:

  1. Issues with file/directory generation are only caught through manual testing
  2. The testing process is time-consuming
  3. Edge cases may be missed due to the manual nature of testing
  4. No automated verification of scaffolding output

Current Test Coverage Limitations

While we have integration tests using tryorama that verify the generated hApp's zome functions work as expected, these tests do not verify the CLI's code generation functionality. This has led to issues:

Recent example: The ui directory was completely omitted due to Crane lib filtering out .hbs files, but existing tests didn't catch this Current tests focus on zome function behavior but don't verify the actual scaffolding output No automated verification of template processing and file generation

Proposed Solution

Implement automated CLI tests in our CI pipeline to:

  • Verify correct file/directory generation
  • Ensure scaffolding creates expected structure
  • Validate command outputs
  • Catch regressions early

Scope

Phase 1: Basic CLI Testing

  • [ ] Set up CLI tests
  • [ ] Implement tests for basic file/directory generation
  • [ ] Add assertions for common scaffolding patterns
  • [ ] Verify command execution results
  • [ ] Add tests for error cases and edge scenarios

Phase 2 (Optional): UI Snapshot Testing

  • [ ] #421
  • [ ] Set up snapshot comparison in CI

Technical Details

Test Coverage Should Include

  • Directory structure validation
  • File content verification
  • Command execution flows
  • Error handling scenarios

Example Test Cases

- Verify scaffold command creates expected directory structure
- Validate file permissions and contents
- Test command option combinations
- Verify error messages for invalid inputs

Success Criteria

  • [ ] All CLI commands have automated tests
  • [ ] CI pipeline includes CLI testing
  • [ ] Reduced time spent on manual testing
  • [ ] Improved confidence in CLI functionality
  • [ ] Documentation for writing new CLI tests

c12i avatar Oct 23 '24 15:10 c12i

Prioritizing #421

c12i avatar Nov 28 '24 16:11 c12i