Fewer flakes with max-cases 1?

[taylordowns2000]

Checking CI's response here:

1 for 1 baby! here we go!

2 for 2!!!

(Watch this space)

[codecov[bot]]

Codecov Report

All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 91.47%. Comparing base (7a57071) to head (8c916b7). Report is 26 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #3150      +/-   ##
==========================================
+ Coverage   91.44%   91.47%   +0.02%     
==========================================
  Files         357      357              
  Lines       13462    13462              
==========================================
+ Hits        12310    12314       +4     
+ Misses       1152     1148       -4     

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

:rocket: New features to boost your workflow:

• :snowflake: Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• :package: JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[stuartc]

@taylordowns2000 this would make the tests much slower, and this isn't how we run tests locally. If anything, we can try changing the ERL_FLAGS to have "less cpu". Right now it's 4:4 I think?

[taylordowns2000]

Understood. I'm up for whatever. If this reduces flakes by 5% though, I'd be happy increasing test runtime by 10x. My workflow is:

1. write code
2. run tests locally
3. push to github
4. request review
5. do entirely unrelated thing (wait however long, doesn't matter as i'm already long gone)
6. retry
7. do entirely unrelated thing (wait however long, doesn't matter as i'm already long gone)
8. repeat...

So the duration doesn't impact me, but each flake costs me about 10 minutes of switching cost/distraction impact.

On Wed, May 7, 2025 at 1:54 PM Stuart Corbishley @.***> wrote:

stuartc left a comment (OpenFn/lightning#3150) https://github.com/OpenFn/lightning/pull/3150#issuecomment-2858296310

@taylordowns2000 https://github.com/taylordowns2000 this would make the tests much slower, and this isn't how we run tests locally. If anything, we can try changing the ERL_FLAGS to have "less cpu". Right now it's 4:4 I think?

— Reply to this email directly, view it on GitHub https://github.com/OpenFn/lightning/pull/3150#issuecomment-2858296310, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACCUBLNZJGR7HLFYHOSC2HL25HYADAVCNFSM6AAAAAB33OWWL6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDQNJYGI4TMMZRGA . You are receiving this because you were mentioned.Message ID: @.***>

-- Taylor Downs CEO, OpenFn https://www.openfn.org

[stuartc]

I hear you. I'd like to do better.

When the elixir upgrade branch lands I'm going to change the test command to:

mix test || mix test --failed

That will rerun just the failed tests, and assuming this doesn't cause some problems with codecov (it might) we will had that "second go around" automated.

[taylordowns2000]

oooooh wow! that's such a cool idea. like... it's an automatic "try just once more before bugging me" line?!

[taylordowns2000]

Hey should I close this PR @stuartc ? You're welcome to if no longer useful