replicate-javascript icon indicating copy to clipboard operation
replicate-javascript copied to clipboard

Bun integration test is flakey, possibly due to bug in `stream` implementation

Open mattt opened this issue 1 year ago • 2 comments

The integration test for Bun fails sporadically with a timeout. Re-running usually results in a successful run.

For example, here's a run that failed: https://github.com/replicate/replicate-javascript/actions/runs/8340326500/job/22824063223

And here's the retry, which succeeded: https://github.com/replicate/replicate-javascript/actions/runs/8340326500/job/22824613649

What's concerning is that both predictions succeeded:

  • First: https://replicate.com/p/n2h3hilb7s55wccy5f4qif53z4
  • Second: https://replicate.com/p/qvsajjdbhkhfr2zlgdxziae4ja

This makes me think that the flakiness is actually a problem in our streaming code. And if we aren't seeing this with the other integration tests, then that suggests a problem in our TextDecoderStream polyfill.

mattt avatar Mar 19 '24 09:03 mattt

@aron Any ideas why we're seeing this?

mattt avatar Mar 19 '24 09:03 mattt

Not sure. There's not much to that TextDecoderStream polyfill we added. Could it also be something to do with the Bun fetch/Response implementation…? We'd have to add more profiling/debugging information to the bun test I think to understand where in the code it's timing out.

aron avatar May 02 '24 11:05 aron

I haven't observed this test flakiness recently — in fact, working on #281, I noticed that Bun was the only integration test that didn't occasionally fail with a 503 error. Switching everything to replicate/canary in #283 should stop this from happening in the future.

mattt avatar Jul 05 '24 13:07 mattt