Flaky test

[lishaduck]

Moving this thread so I don't lose track of it: ↓

I've noticed this there's a test that's been flaking ("Running using other configuration (without errors)"):

-- UNEXPECTED ERROR ------------------------------------------------------------

I ran into an unexpected error. Please open an issue at the following link:
  https://github.com/jfmengels/node-elm-review/issues/new

Please include this error message and as much detail as you can provide. Running
with --debug might give additional information. If you can, please provide a
setup that makes it easy to reproduce the error. That will make it much easier
to fix the issue.

Below is the error that was encountered.
--------------------------------------------------------------------------------
<local-path>/test/project-with-errors/elm-stuff/generated-code/jfmengels/elm-review/cli/<version>/review-applications/bd2a2bb6c965dac5241ce8c4d0615d83.js:10328
    		$temp$offset = offset + 
    						                    

SyntaxError: Unexpected end of input

  at loadCompiledElmApp (<local-path>/lib/load-compiled-app.js:28:18)
  at initWithoutWorker (<local-path>/lib/app-wrapper.js:144:21)
  "
  at Object.test (test/review.test.js:37:1)

Originally posted by @lishaduck in https://github.com/jfmengels/node-elm-review/issues/308#issuecomment-2467097204

[lishaduck]

I've noticed this there's a test that's been flaking ("Running using other configuration (without errors)"):

Yes, I've noticed the same thing. I had a look, thinking it could be us not waiting for the compilation to be finalized, but the code looks fine. I still think it's the writing of the file (or the loading of it, but that's weirder) that is getting cut off early somehow, but I haven't figured out how yet. :thinking:

It sounds pretty recent though. I don't think we saw this one before, but I could be wrong.

Originally posted by @jfmengels in https://github.com/jfmengels/node-elm-review/issues/308#issuecomment-2467667152

[lishaduck]

It sounds pretty recent though. I don't think we saw this one before, but I could be wrong.

Well, it's really started flaking. Will look into it. For reference, here's the first failure from it: https://github.com/jfmengels/node-elm-review/actions/runs/11769034390/job/32779541170

EDIT: Commit-wise, there was a different flake on #306 (it's older than my work here, I think I remember seeing it a while ago), and then fs-extra (#303) seems to have started this flake.

EDIT 2: Am gonna try to see if I reproduce it before #303.

Originally posted by @lishaduck in https://github.com/jfmengels/node-elm-review/issues/308#issuecomment-2468229277

[lishaduck]

Was able to repro with 38453c161ca3c4273482760b7c4576203e3ced72, so sadly not #303.

Originally posted by @lishaduck in https://github.com/jfmengels/node-elm-review/issues/308#issuecomment-2468394493

[lishaduck]

Huh. I can't repro on #304 for the life of me,[^1] so I guess it's gotta be #306 (or, I suppose, #305), which makes no sense.

[^1]: I've run tests ~~10+~~ 9 times on the commit, to no avail.

Originally posted by @lishaduck in https://github.com/jfmengels/node-elm-review/issues/308#issuecomment-2468411960

[lishaduck]

@jfmengels, any ideas?

Originally posted by @lishaduck in https://github.com/jfmengels/node-elm-review/issues/308#issuecomment-2469290010

[jfmengels]

I haven't had the time to delve into it more, so my thoughts are still the same as what is written above.

Do you have the power to re-run CI runs by the way or is it only me? (Because if you can, then this is annoying but more easily bypassable).

[lishaduck]

Do you have the power to re-run CI runs by the way or is it only me? (Because if you can, then this is annoying but more easily bypassable).

No, I don't unfortunately (surprisingly, GitHub has a single secure default for actions 🎉 🤣).

[lishaduck]

What? Another flake: https://github.com/jfmengels/node-elm-review/actions/runs/11902573547/job/33167861766

This time in "should retrieve elm binary from PATH":

"-- UNEXPECTED ERROR ------------------------------------------------------------

I ran into an unexpected error. Please open an issue at the following link:
  https://github.com/jfmengels/node-elm-review/issues/new

Please include this error message and as much detail as you can provide. Running
with --debug might give additional information. If you can, please provide a
setup that makes it easy to reproduce the error. That will make it much easier
to fix the issue.

Below is the error that was encountered.
--------------------------------------------------------------------------------
TypeError: Cannot read properties of undefined (reading 'Elm')

  at initWithoutWorker (<local-path>/lib/app-wrapper.js:[145](https://github.com/jfmengels/node-elm-review/actions/runs/11902573547/job/33167861766#step:10:152):29)
  at Object.init (<local-path>/lib/app-wrapper.js:25:10)
  at Object.initializeApp (<local-path>/lib/runner.js:112:26)
  at <local-path>/lib/main.js:81:27
      at async Promise.all (index 0)
  at async runElmReview (<local-path>/lib/main.js:79:19)
  at async main (<local-path>/lib/main.js:271:5)
  "
  at Object.test (test/compiler-flag.test.js:13:1)

I just noticed that we've got two stacktraces here. Will need to keep that in mind.

It makes me think of 8fbe6f62a41e9c46d9529b5724b15ad443788111, but that didn't change any runtime code.

[lishaduck]

I'll try to get to this this week. This was frustrating me, so I took a break to start a "small" side project to help make TS more Elmy, but the scope of that is frustrating me as well, so I'm back :)

[lishaduck]

Oh, fixed by #322 (hopefully).

[lishaduck]

Oh, reopening this as we hit this again, unfortunately. It is a bit better though!

[lishaduck]

Dagumit. Even after #325, it's still running into issues. Perhaps it's a stale cache in ci? It shouldn't invoking elm-review in parallel anymore. (Wait, does Jest run in parallel? yup).

[jfmengels]

I've cleared the cache just in case :shrug:

[lishaduck]

I've cleared the cache just in case 🤷

Can you rerun https://github.com/jfmengels/node-elm-review/actions/runs/12214733248 to see if it helped?

[jfmengels]

It passed this time.

[lishaduck]

Hopefully that was it 🤞

[lishaduck]

Nope, flaked on #317.

[lishaduck]

Heyyyyyy! Post-#327, we just hit outdated snapshots and rate limits, but we didn't hit the flake!

If it comes back, I'll reopen it, but I'm cautiously optimistic it's fixed (enough) and #324 should cover the rest.

[lishaduck]

CI looks like it's considerably less flakey now? I'm still having trouble reproing this locally, so I guess I'll close this as it's not really actionable 🤷‍♂

[lishaduck]

Certainly not perfect, but 4/5ish passing seems good enough.

[lishaduck]

Keep forgetting to reopen this.

We had a nice run of it for a day or two. Hmm.