website icon indicating copy to clipboard operation
website copied to clipboard

fix: flaky CI tests by isolating temp directories and preventing side effects

Open Sourya07 opened this issue 4 weeks ago • 5 comments

There is three primary causes for the race conditions and flakes.

1.Shared Resource Collision 2.Unintended Side-Effects 3.Source Tree Pollution Key fixes: • Use unique temp directories (fs.mkdtempSync) in tests to avoid collisions. • Move test-generated files out of the source tree and into os.tmpdir(). • Prevent scripts/index.ts from executing start() on import by running it only in CLI mode.

Ran tests locally withnpx jest tests/build-newsroom-videos.test.ts tests/build-tools.test.ts tests/build-meetings.test.ts tests/index.test.tsmultiple times. Verified all tests pass consistently without collision.

issue no -> #4732

Summary by CodeRabbit

  • Updates

    • Corrected technology label to "AsyncAPI CLI" in API directory and code generator listings
  • Tests

    • Enhanced test isolation using temporary directories instead of hardcoded paths
    • Improved module execution behavior to activate only when invoked directly

✏️ Tip: You can customize this high-level summary in your review settings.

Sourya07 avatar Dec 19 '25 06:12 Sourya07

Deploy Preview for asyncapi-website ready!

Built without sensitive environment variables

Name Link
Latest commit 8086f6fec5f942be02ffbdb14c61e2531fb05f0c
Latest deploy log https://app.netlify.com/projects/asyncapi-website/deploys/6944ea82e061430008dc65ad
Deploy Preview https://deploy-preview-4743--asyncapi-website.netlify.app
Preview on mobile
Toggle QR Code...

QR Code

Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify project configuration.

netlify[bot] avatar Dec 19 '25 06:12 netlify[bot]

Walkthrough

Configuration label updated from "Liquid" to "AsyncAPI CLI" in two locations. Module execution in scripts now guarded by main module check. Test files refactored to use temporary directories instead of hardcoded paths for output and configuration files.

Changes

Cohort / File(s) Summary
Configuration labeling
config/tools.json
Renamed technology label "Liquid" to "AsyncAPI CLI" in AsyncAPI-Directory filters and ZenWave SDK Code Generators sections
Module entry point
scripts/index.ts
Added conditional module execution guard to invoke start() only when file is executed directly (import.meta.url check); minor whitespace adjustment
Test infrastructure refactoring
tests/build-meetings.test.ts
tests/build-newsroom-videos.test.ts
tests/build-tools.test.ts
Migrated test setup from hardcoded paths to runtime-generated temporary directories using mkdtempSync; moved path initialization from const to let variables in beforeAll/afterAll hooks; added file system setup with ensureDirSync and initial fixture writes; enhanced cleanup logic

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~20 minutes

  • Test file refactoring patterns: Verify consistent temp directory creation and cleanup across three test files; ensure mkdtempSync and path resolution are correctly applied in each context
  • Module guard condition: Confirm the import.meta.url check correctly identifies direct execution vs. module import
  • Test isolation: Ensure beforeAll/afterAll hooks properly initialize and clean up temporary directories without side effects between tests
  • Configuration label change: Verify label rename is applied consistently in both locations and doesn't affect API contracts

Poem

🐰 From "Liquid" flows to CLI's bright gleam, Tests now hop through temp dirs, a cleaner dream, Guards watch the entry, imports run free, Cleanup assured—tidy files, you'll see! 🌟

Pre-merge checks and finishing touches

✅ Passed checks (3 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The PR title accurately summarizes the main change: fixing flaky CI tests through temp directory isolation and preventing side effects, which directly corresponds to the primary objectives.
Docstring Coverage ✅ Passed Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%.
✨ Finishing touches
  • [ ] 📝 Generate docstrings
🧪 Generate unit tests (beta)
  • [ ] Create PR with unit tests
  • [ ] Post copyable unit tests in a comment

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

coderabbitai[bot] avatar Dec 19 '25 06:12 coderabbitai[bot]

Codecov Report

:x: Patch coverage is 0% with 2 lines in your changes missing coverage. Please review. :white_check_mark: Project coverage is 99.75%. Comparing base (3a1ca24) to head (8086f6f). :warning: Report is 7 commits behind head on master.

Files with missing lines Patch % Lines
scripts/index.ts 0.00% 1 Missing and 1 partial :warning:
Additional details and impacted files
@@             Coverage Diff             @@
##            master    #4743      +/-   ##
===========================================
- Coverage   100.00%   99.75%   -0.25%     
===========================================
  Files           22       22              
  Lines          799      800       +1     
  Branches       146      147       +1     
===========================================
- Hits           799      798       -1     
- Misses           0        1       +1     
- Partials         0        1       +1     

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

:rocket: New features to boost your workflow:
  • :snowflake: Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • :package: JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

codecov[bot] avatar Dec 19 '25 06:12 codecov[bot]

⚡️ Lighthouse report for the changes in this PR:

Category Score
🔴 Performance 36
🟢 Accessibility 98
🟢 Best practices 92
🟢 SEO 100
🔴 PWA 33

Lighthouse ran on https://deploy-preview-4743--asyncapi-website.netlify.app/

asyncapi-bot avatar Dec 19 '25 06:12 asyncapi-bot

@vishvamsinh28 take a look on this . Regarding Codecov failure: The coverage drop in

scripts/index.ts is intentional and expected. Previously, the start() function was executing every time the file was imported by the test suite ( tests/index.test.ts ), which caused the script to run against the real filesystem in the background. This was a major source of flakiness.

I wrapped

start() in a conditional check (if (process.argv[1] === ...)), ensuring it only runs when the script is executed directly (e.g., npm run build), and never when imported by tests. This correctly stops the code from running during tests, which Codecov flags as "missing coverage," but it is actually the fix itself.

Sourya07 avatar Dec 19 '25 06:12 Sourya07

Already a work in progress by someone else

princerajpoot20 avatar Dec 23 '25 17:12 princerajpoot20