fix: flaky CI tests by isolating temp directories and preventing side effects
There is three primary causes for the race conditions and flakes.
1.Shared Resource Collision 2.Unintended Side-Effects 3.Source Tree Pollution Key fixes: • Use unique temp directories (fs.mkdtempSync) in tests to avoid collisions. • Move test-generated files out of the source tree and into os.tmpdir(). • Prevent scripts/index.ts from executing start() on import by running it only in CLI mode.
Ran tests locally withnpx jest tests/build-newsroom-videos.test.ts tests/build-tools.test.ts tests/build-meetings.test.ts tests/index.test.tsmultiple times.
Verified all tests pass consistently without collision.
issue no -> #4732
Summary by CodeRabbit
-
Updates
- Corrected technology label to "AsyncAPI CLI" in API directory and code generator listings
-
Tests
- Enhanced test isolation using temporary directories instead of hardcoded paths
- Improved module execution behavior to activate only when invoked directly
✏️ Tip: You can customize this high-level summary in your review settings.
Deploy Preview for asyncapi-website ready!
Built without sensitive environment variables
| Name | Link |
|---|---|
| Latest commit | 8086f6fec5f942be02ffbdb14c61e2531fb05f0c |
| Latest deploy log | https://app.netlify.com/projects/asyncapi-website/deploys/6944ea82e061430008dc65ad |
| Deploy Preview | https://deploy-preview-4743--asyncapi-website.netlify.app |
| Preview on mobile | Toggle QR Code...Use your smartphone camera to open QR code link. |
To edit notification comments on pull requests, go to your Netlify project configuration.
Walkthrough
Configuration label updated from "Liquid" to "AsyncAPI CLI" in two locations. Module execution in scripts now guarded by main module check. Test files refactored to use temporary directories instead of hardcoded paths for output and configuration files.
Changes
| Cohort / File(s) | Summary |
|---|---|
Configuration labeling config/tools.json |
Renamed technology label "Liquid" to "AsyncAPI CLI" in AsyncAPI-Directory filters and ZenWave SDK Code Generators sections |
Module entry point scripts/index.ts |
Added conditional module execution guard to invoke start() only when file is executed directly (import.meta.url check); minor whitespace adjustment |
Test infrastructure refactoring tests/build-meetings.test.ts tests/build-newsroom-videos.test.ts tests/build-tools.test.ts |
Migrated test setup from hardcoded paths to runtime-generated temporary directories using mkdtempSync; moved path initialization from const to let variables in beforeAll/afterAll hooks; added file system setup with ensureDirSync and initial fixture writes; enhanced cleanup logic |
Estimated code review effort
🎯 3 (Moderate) | ⏱️ ~20 minutes
- Test file refactoring patterns: Verify consistent temp directory creation and cleanup across three test files; ensure mkdtempSync and path resolution are correctly applied in each context
- Module guard condition: Confirm the import.meta.url check correctly identifies direct execution vs. module import
- Test isolation: Ensure beforeAll/afterAll hooks properly initialize and clean up temporary directories without side effects between tests
- Configuration label change: Verify label rename is applied consistently in both locations and doesn't affect API contracts
Poem
🐰 From "Liquid" flows to CLI's bright gleam, Tests now hop through temp dirs, a cleaner dream, Guards watch the entry, imports run free, Cleanup assured—tidy files, you'll see! 🌟
Pre-merge checks and finishing touches
✅ Passed checks (3 passed)
| Check name | Status | Explanation |
|---|---|---|
| Description Check | ✅ Passed | Check skipped - CodeRabbit’s high-level summary is enabled. |
| Title check | ✅ Passed | The PR title accurately summarizes the main change: fixing flaky CI tests through temp directory isolation and preventing side effects, which directly corresponds to the primary objectives. |
| Docstring Coverage | ✅ Passed | Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%. |
✨ Finishing touches
- [ ] 📝 Generate docstrings
🧪 Generate unit tests (beta)
- [ ] Create PR with unit tests
- [ ] Post copyable unit tests in a comment
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.
Comment @coderabbitai help to get the list of available commands and usage tips.
Codecov Report
:x: Patch coverage is 0% with 2 lines in your changes missing coverage. Please review.
:white_check_mark: Project coverage is 99.75%. Comparing base (3a1ca24) to head (8086f6f).
:warning: Report is 7 commits behind head on master.
| Files with missing lines | Patch % | Lines |
|---|---|---|
| scripts/index.ts | 0.00% | 1 Missing and 1 partial :warning: |
Additional details and impacted files
@@ Coverage Diff @@
## master #4743 +/- ##
===========================================
- Coverage 100.00% 99.75% -0.25%
===========================================
Files 22 22
Lines 799 800 +1
Branches 146 147 +1
===========================================
- Hits 799 798 -1
- Misses 0 1 +1
- Partials 0 1 +1
:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.
:rocket: New features to boost your workflow:
- :snowflake: Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
- :package: JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
⚡️ Lighthouse report for the changes in this PR:
| Category | Score |
|---|---|
| 🔴 Performance | 36 |
| 🟢 Accessibility | 98 |
| 🟢 Best practices | 92 |
| 🟢 SEO | 100 |
| 🔴 PWA | 33 |
Lighthouse ran on https://deploy-preview-4743--asyncapi-website.netlify.app/
@vishvamsinh28 take a look on this . Regarding Codecov failure: The coverage drop in
scripts/index.ts is intentional and expected. Previously, the start() function was executing every time the file was imported by the test suite ( tests/index.test.ts ), which caused the script to run against the real filesystem in the background. This was a major source of flakiness.
I wrapped
start() in a conditional check (if (process.argv[1] === ...)), ensuring it only runs when the script is executed directly (e.g., npm run build), and never when imported by tests. This correctly stops the code from running during tests, which Codecov flags as "missing coverage," but it is actually the fix itself.
Already a work in progress by someone else