Fallback service failed to fetch module; exception = kj/async-io-unix.c++:1524: disconnected: connect(): Connection refused
What versions & operating system are you using?
System: OS: macOS 15.6.1 CPU: (16) arm64 Apple M4 Max Memory: 1.15 GB / 48.00 GB Shell: 5.9 - /bin/zsh Binaries: Node: 20.12.1 - /Users/dmaretskyi/.proto/shims/node Yarn: 1.22.22 - /Users/dmaretskyi/.npm-global/bin/yarn npm: 10.5.0 - /Users/dmaretskyi/.proto/tools/node/20.12.1/bin/npm Deno: 2.1.10 - /opt/homebrew/bin/deno npmPackages: @cloudflare/vitest-pool-workers: ^0.8.19 => 0.8.71 wrangler: ^4.45.0 => 4.45.0
Please provide a link to a minimal reproduction
https://github.com/dmaretskyi/vpw-repro/tree/master
Describe the Bug
Re-openning https://github.com/cloudflare/workers-sdk/issues/8853 with repro.
The linked repository has a reproduction script to run. It's difficult to pinpoint exactly what is causing the bug, since it seems to be a combinator of factors that are required to trigger this:
- Running vitest-pool-workers multiple times repeatedly (I'm suspecting some system-global state gets corrupted).
- Having multiple separate test-suite files.
- Having a worker with large bundle size (I've imported couple open-source libs we use to simulate this).
In our large production codebase the vitest-pool-workers fails when run 2-3 times in a short sequence, with this repro it might take 5-10 iterations.
Please provide any relevant error logs
workerd/server/fallback-service.c++:158: error: Fallback service failed to fetch module; exception = kj/async-io-unix.c++:1538: disconnected: connect(): Connection refused
stack: 1047f0f2b 1047f1103 1047bf3db 1047eb5ac 102230be0 102ef0638 1047e2bb4 1047cc110 10330f003 10330f9a3 1033118c7 1032f5448 1032fb018 1025acb03 1025ac24b; spec = /?specifier=%2FUsers%2Fdmaretskyi%2FCode%2Fdxos%2Fvpw-repro%2Fnode_modules%2F.pnpm%2Feffect%403.18.4%2Fnode_modules%2Feffect%2Fdist%2Fesm%2FTuple.js&referrer=%2FUsers%2Fdmaretskyi%2FCode%2Fdxos%2Fvpw-repro%2Fnode_modules%2F.pnpm%2Feffect%403.18.4%2Fnode_modules%2Feffect%2Fdist%2Fesm%2Findex.js&rawSpecifier=.%2FTuple.js
workerd/server/fallback-service.c++:158: error: Fallback service failed to fetch module; exception = kj/async-io-unix.c++:1538: disconnected: connect(): Connection refused
stack: 1047f0f2b 1047f1103 1047bf3db 1047eb5ac 102230be0 102ef0638 1047e2bb4 1047cc110 10330f003 10330f9a3 1033118c7 1032f5448 1032fb018 1025acb03 1025ac24b; spec = /?specifier=%2FUsers%2Fdmaretskyi%2FCode%2Fdxos%2Fvpw-repro%2Fnode_modules%2F.pnpm%2Feffect%403.18.4%2Fnode_modules%2Feffect%2Fdist%2Fesm%2Findex.js&referrer=%2FUsers%2Fdmaretskyi%2FCode%2Fdxos%2Fvpw-repro%2Fnode_modules%2F.pnpm%2Fvite-node%403.2.4%2Fnode_modules%2Fvite-node%2Fdist%2Fclient.mjs&rawSpecifier=%2FUsers%2Fdmaretskyi%2FCode%2Fdxos%2Fvpw-repro%2Fnode_modules%2F.pnpm%2Feffect%403.18.4%2Fnode_modules%2Feffect%2Fdist%2Fesm%2Findex.js
workerd/server/fallback-service.c++:158: error: Fallback service failed to fetch module; exception = kj/async-io-unix.c++:1538: disconnected: connect(): Connection refused
stack: 1047f0f2b 1047f1103 1047bf3db 1047eb5ac 102230be0 102ef0638 1047e2bb4 1047cc110 10330f003 10330f9a3 1033118c7 1032f5448 1032fb018 1025acb03 1025ac24b; spec = /?specifier=%2FUsers%2Fdmaretskyi%2FCode%2Fdxos%2Fvpw-repro%2Fnode_modules%2F.pnpm%2Feffect%403.18.4%2Fnode_modules%2Feffect%2Fdist%2Fesm%2Findex.js&referrer=%2FUsers%2Fdmaretskyi%2FCode%2Fdxos%2Fvpw-repro%2Fnode_modules%2F.pnpm%2Fvite-node%403.2.4%2Fnode_modules%2Fvite-node%2Fdist%2Fclient.mjs&rawSpecifier=%2FUsers%2Fdmaretskyi%2FCode%2Fdxos%2Fvpw-repro%2Fnode_modules%2F.pnpm%2Feffect%403.18.4%2Fnode_modules%2Feffect%2Fdist%2Fesm%2Findex.js
workerd/server/fallback-service.c++:158: error: Fallback service failed to fetch module; exception = kj/async-io-unix.c++:1538: disconnected: connect(): Connection refused
stack: 1047f0f2b 1047f1103 1047bf3db 1047eb5ac 102230be0 102ef0638 1047e2bb4 1047cc110 10330f003 10330f9a3 1033118c7 1032f5448 1032fb018 1025acb03 1025ac24b; spec = /?specifier=%2FUsers%2Fdmaretskyi%2FCode%2Fdxos%2Fvpw-repro%2Fnode_modules%2F.pnpm%2Feffect%403.18.4%2Fnode_modules%2Feffect%2Fdist%2Fesm%2Findex.js&referrer=%2FUsers%2Fdmaretskyi%2FCode%2Fdxos%2Fvpw-repro%2Fnode_modules%2F.pnpm%2Fvite-node%403.2.4%2Fnode_modules%2Fvite-node%2Fdist%2Fclient.mjs&rawSpecifier=%2FUsers%2Fdmaretskyi%2FCode%2Fdxos%2Fvpw-repro%2Fnode_modules%2F.pnpm%2Feffect%403.18.4%2Fnode_modules%2Feffect%2Fdist%2Fesm%2Findex.js
workerd/jsg/util.c++:393: error: e = kj/async-io-unix.c++:1523: failed: connect(): Can't assign requested address; address = 127.0.0.1:54089
stack: 1047ed1b3 1047eaf6f 1047eb5ac 102230be0 102260263 102ef0638 1047e2bb4 1047cc110 10330f003 10330f9a3 1033118c7 1032f5448 1032fb018 10330c32c 1021f4230 102cd883c 102bc8ff4; sentryErrorContext = jsgInternalError; wdErrId = tqve6n0rm9pjquorg008th6i
workerd/io/worker.c++:2168: info: uncaught exception; source = Uncaught (in promise); stack = Error: internal error; reference = tqve6n0rm9pjquorg008th6i
workerd/io/io-context.c++:389: info: uncaught exception; exception = workerd/jsg/_virtual_includes/iterator/workerd/jsg/value.h:1532: failed: jsg.Error: internal error; reference = tqve6n0rm9pjquorg008th6i
stack: 102c95a48 102c96613 1047bf3db 102acad8c 102acb178
Error running worker: Error: internal error; reference = tqve6n0rm9pjquorg008th6i
at WorkersSnapshotEnvironment.readSnapshotFile (Users/dmaretskyi/Code/dxos/vpw-repro/node_modules/.pnpm/@[email protected]_@[email protected]_@[email protected][email protected]/node_modules/@cloudflare/vitest-pool-workers/dist/worker/lib/cloudflare/test-runner.mjs:45:17)
at SnapshotState.create (Users/dmaretskyi/Code/dxos/vpw-repro/node_modules/.pnpm/@[email protected]/node_modules/@vitest/snapshot/dist/index.js:1926:19)
at SnapshotClient.setup (Users/dmaretskyi/Code/dxos/vpw-repro/node_modules/.pnpm/@[email protected]/node_modules/@vitest/snapshot/dist/index.js:2220:39)
at WorkersTestRunner.onBeforeRunSuite (Users/dmaretskyi/Code/dxos/vpw-repro/node_modules/.pnpm/[email protected]/node_modules/vitest/dist/runners.js:182:53)
at runSuite (Users/dmaretskyi/Code/dxos/vpw-repro/node_modules/.pnpm/@[email protected]/node_modules/@vitest/runner/dist/chunk-hooks.js:1683:2)
at runFiles (Users/dmaretskyi/Code/dxos/vpw-repro/node_modules/.pnpm/@[email protected]/node_modules/@vitest/runner/dist/chunk-hooks.js:1787:3)
at startTests (Users/dmaretskyi/Code/dxos/vpw-repro/node_modules/.pnpm/@[email protected]/node_modules/@vitest/runner/dist/chunk-hooks.js:1820:3)
at Users/dmaretskyi/Code/dxos/vpw-repro/node_modules/.pnpm/[email protected]/node_modules/vitest/dist/chunks/runBaseTests.9Ij9_de-.js:117:26
at withEnv (Users/dmaretskyi/Code/dxos/vpw-repro/node_modules/.pnpm/[email protected]/node_modules/vitest/dist/chunks/runBaseTests.9Ij9_de-.js:84:3)
at run (Users/dmaretskyi/Code/dxos/vpw-repro/node_modules/.pnpm/[email protected]/node_modules/vitest/dist/chunks/runBaseTests.9Ij9_de-.js:109:2)
workerd/jsg/util.c++:393: error: e = kj/async-io-unix.c++:1523: failed: connect(): Can't assign requested address; address = 127.0.0.1:54089
stack: 1047ed1b3 1047eaf6f 1047eb5ac 102230be0 102260263 102ef0638 1047e2bb4 1047cc110 10330f003 10330f9a3 1033118c7 1032f5448 1032fb018 10330c32c 1021f4230 102cd883c 102bc8ff4; sentryErrorContext = jsgInternalError; wdErrId = m7h7gif5b8q0nfj0iqbbaqvl
workerd/io/worker.c++:2168: info: uncaught exception; source = Uncaught (in promise); stack = Error: internal error; reference = m7h7gif5b8q0nfj0iqbbaqvl
workerd/io/io-context.c++:389: info: uncaught exception; exception = workerd/jsg/_virtual_includes/iterator/workerd/jsg/value.h:1532: failed: jsg.Error: internal error; reference = m7h7gif5b8q0nfj0iqbbaqvl
stack: 102c95a48 102c96613 1047bf3db 102acad8c 102acb178
Error running worker: Error: internal error; reference = m7h7gif5b8q0nfj0iqbbaqvl
at WorkersSnapshotEnvironment.readSnapshotFile (Users/dmaretskyi/Code/dxos/vpw-repro/node_modules/.pnpm/@[email protected]_@[email protected]_@[email protected][email protected]/node_modules/@cloudflare/vitest-pool-workers/dist/worker/lib/cloudflare/test-runner.mjs:45:17)
at SnapshotState.create (Users/dmaretskyi/Code/dxos/vpw-repro/node_modules/.pnpm/@[email protected]/node_modules/@vitest/snapshot/dist/index.js:1926:19)
at SnapshotClient.setup (Users/dmaretskyi/Code/dxos/vpw-repro/node_modules/.pnpm/@[email protected]/node_modules/@vitest/snapshot/dist/index.js:2220:39)
at WorkersTestRunner.onBeforeRunSuite (Users/dmaretskyi/Code/dxos/vpw-repro/node_modules/.pnpm/[email protected]/node_modules/vitest/dist/runners.js:182:53)
at runSuite (Users/dmaretskyi/Code/dxos/vpw-repro/node_modules/.pnpm/@[email protected]/node_modules/@vitest/runner/dist/chunk-hooks.js:1683:2)
at runFiles (Users/dmaretskyi/Code/dxos/vpw-repro/node_modules/.pnpm/@[email protected]/node_modules/@vitest/runner/dist/chunk-hooks.js:1787:3)
at startTests (Users/dmaretskyi/Code/dxos/vpw-repro/node_modules/.pnpm/@[email protected]/node_modules/@vitest/runner/dist/chunk-hooks.js:1820:3)
at Users/dmaretskyi/Code/dxos/vpw-repro/node_modules/.pnpm/[email protected]/node_modules/vitest/dist/chunks/runBaseTests.9Ij9_de-.js:117:26
at withEnv (Users/dmaretskyi/Code/dxos/vpw-repro/node_modules/.pnpm/[email protected]/node_modules/vitest/dist/chunks/runBaseTests.9Ij9_de-.js:84:3)
at run (Users/dmaretskyi/Code/dxos/vpw-repro/node_modules/.pnpm/[email protected]/node_modules/vitest/dist/chunks/runBaseTests.9Ij9_de-.js:109:2)
workerd/server/fallback-service.c++:158: error: Fallback service failed to fetch module; exception = kj/async-io-unix.c++:1538: disconnected: connect(): Connection refused
stack: 1047f0f2b 1047f1103 1047bf3db 1047eb5ac 102230be0 102ef0638 1047e2bb4 1047cc110 10330f003 10330f9a3 1033118c7 1032f5448 1032fb018 1025acb03 1025ac24b; spec = /?specifier=%2FUsers%2Fdmaretskyi%2FCode%2Fdxos%2Fvpw-repro%2Fnode_modules%2F.pnpm%2F%40effect%2Bai-openai%400.34.0_%40effect%2Bai%400.31.1_%40effect%2Bexperimental%400.56.0_%40effect%2Bplatform%400.92._hzv5wf6kwlojwx3xitzxy4qiqu%2Fnode_modules%2F%40effect%2Fai-openai%2Fdist%2Fesm%2Findex.js&referrer=%2FUsers%2Fdmaretskyi%2FCode%2Fdxos%2Fvpw-repro%2Fnode_modules%2F.pnpm%2Fvite-node%403.2.4%2Fnode_modules%2Fvite-node%2Fdist%2Fclient.mjs&rawSpecifier=%2FUsers%2Fdmaretskyi%2FCode%2Fdxos%2Fvpw-repro%2Fnode_modules%2F.pnpm%2F%40effect%2Bai-openai%400.34.0_%40effect%2Bai%400.31.1_%40effect%2Bexperimental%400.56.0_%40effect%2Bplatform%400.92._hzv5wf6kwlojwx3xitzxy4qiqu%2Fnode_modules%2F%40effect%2Fai-openai%2Fdist%2Fesm%2Findex.js
workerd/jsg/util.c++:393: error: e = kj/async-io-unix.c++:1523: failed: connect(): Can't assign requested address; address = 127.0.0.1:54089
stack: 1047ed1b3 1047eaf6f 1047eb5ac 102230be0 102260263 102ef0638 1047e2bb4 1047cc110 10330f003 10330f9a3 1033118c7 1032f5448 1032fb018 10330c32c 1021f4230 102cd883c 102bc8ff4; sentryErrorContext = jsgInternalError; wdErrId = uiutcsne1oif3n9rqmd02a4u
workerd/io/worker.c++:2168: info: uncaught exception; source = Uncaught (in promise); stack = Error: internal error; reference = uiutcsne1oif3n9rqmd02a4u
workerd/io/io-context.c++:389: info: uncaught exception; exception = workerd/jsg/_virtual_includes/iterator/workerd/jsg/value.h:1532: failed: jsg.Error: internal error; reference = uiutcsne1oif3n9rqmd02a4u
stack: 102c95a48 102c96613 1047bf3db 102acad8c 102acb178
workerd/jsg/util.c++:393: error: e = kj/async-io-unix.c++:1523: failed: connect(): Can't assign requested address; address = 127.0.0.1:54089
stack: 1047ed1b3 1047eaf6f 1047eb5ac 102230be0 102260263 102ef0638 1047e2bb4 1047cc110 10330f003 10330f9a3 1033118c7 1032f5448 1032fb018 10330c32c 1021f4230 102cd883c 102bc8ff4; sentryErrorContext = jsgInternalError; wdErrId = sq83d5fdic38i54erjqnivmg
workerd/io/worker.c++:2168: info: uncaught exception; source = Uncaught (in promise); stack = Error: internal error; reference = sq83d5fdic38i54erjqnivmg
workerd/io/io-context.c++:389: info: uncaught exception; exception = workerd/jsg/_virtual_includes/iterator/workerd/jsg/value.h:1532: failed: jsg.Error: internal error; reference = sq83d5fdic38i54erjqnivmg
stack: 102c95a48 102c96613 1047bf3db 102acad8c 102acb178
Error running worker: Error: internal error; reference = uiutcsne1oif3n9rqmd02a4u
at WorkersSnapshotEnvironment.readSnapshotFile (Users/dmaretskyi/Code/dxos/vpw-repro/node_modules/.pnpm/@[email protected]_@[email protected]_@[email protected][email protected]/node_modules/@cloudflare/vitest-pool-workers/dist/worker/lib/cloudflare/test-runner.mjs:45:17)
at SnapshotState.create (Users/dmaretskyi/Code/dxos/vpw-repro/node_modules/.pnpm/@[email protected]/node_modules/@vitest/snapshot/dist/index.js:1926:19)
at SnapshotClient.setup (Users/dmaretskyi/Code/dxos/vpw-repro/node_modules/.pnpm/@[email protected]/node_modules/@vitest/snapshot/dist/index.js:2220:39)
at WorkersTestRunner.onBeforeRunSuite (Users/dmaretskyi/Code/dxos/vpw-repro/node_modules/.pnpm/[email protected]/node_modules/vitest/dist/runners.js:182:53)
at runSuite (Users/dmaretskyi/Code/dxos/vpw-repro/node_modules/.pnpm/@[email protected]/node_modules/@vitest/runner/dist/chunk-hooks.js:1683:2)
at runFiles (Users/dmaretskyi/Code/dxos/vpw-repro/node_modules/.pnpm/@[email protected]/node_modules/@vitest/runner/dist/chunk-hooks.js:1787:3)
at startTests (Users/dmaretskyi/Code/dxos/vpw-repro/node_modules/.pnpm/@[email protected]/node_modules/@vitest/runner/dist/chunk-hooks.js:1820:3)
at Users/dmaretskyi/Code/dxos/vpw-repro/node_modules/.pnpm/[email protected]/node_modules/vitest/dist/chunks/runBaseTests.9Ij9_de-.js:117:26
at withEnv (Users/dmaretskyi/Code/dxos/vpw-repro/node_modules/.pnpm/[email protected]/node_modules/vitest/dist/chunks/runBaseTests.9Ij9_de-.js:84:3)
at run (Users/dmaretskyi/Code/dxos/vpw-repro/node_modules/.pnpm/[email protected]/node_modules/vitest/dist/chunks/runBaseTests.9Ij9_de-.js:109:2)
Error running worker: Error: internal error; reference = sq83d5fdic38i54erjqnivmg
at WorkersSnapshotEnvironment.readSnapshotFile (Users/dmaretskyi/Code/dxos/vpw-repro/node_modules/.pnpm/@[email protected]_@[email protected]_@[email protected][email protected]/node_modules/@cloudflare/vitest-pool-workers/dist/worker/lib/cloudflare/test-runner.mjs:45:17)
at SnapshotState.create (Users/dmaretskyi/Code/dxos/vpw-repro/node_modules/.pnpm/@[email protected]/node_modules/@vitest/snapshot/dist/index.js:1926:19)
at SnapshotClient.setup (Users/dmaretskyi/Code/dxos/vpw-repro/node_modules/.pnpm/@[email protected]/node_modules/@vitest/snapshot/dist/index.js:2220:39)
at WorkersTestRunner.onBeforeRunSuite (Users/dmaretskyi/Code/dxos/vpw-repro/node_modules/.pnpm/[email protected]/node_modules/vitest/dist/runners.js:182:53)
at runSuite (Users/dmaretskyi/Code/dxos/vpw-repro/node_modules/.pnpm/@[email protected]/node_modules/@vitest/runner/dist/chunk-hooks.js:1683:2)
at runFiles (Users/dmaretskyi/Code/dxos/vpw-repro/node_modules/.pnpm/@[email protected]/node_modules/@vitest/runner/dist/chunk-hooks.js:1787:3)
at startTests (Users/dmaretskyi/Code/dxos/vpw-repro/node_modules/.pnpm/@[email protected]/node_modules/@vitest/runner/dist/chunk-hooks.js:1820:3)
at Users/dmaretskyi/Code/dxos/vpw-repro/node_modules/.pnpm/[email protected]/node_modules/vitest/dist/chunks/runBaseTests.9Ij9_de-.js:117:26
at withEnv (Users/dmaretskyi/Code/dxos/vpw-repro/node_modules/.pnpm/[email protected]/node_modules/vitest/dist/chunks/runBaseTests.9Ij9_de-.js:84:3)
at run (Users/dmaretskyi/Code/dxos/vpw-repro/node_modules/.pnpm/[email protected]/node_modules/vitest/dist/chunks/runBaseTests.9Ij9_de-.js:109:2)
workerd/jsg/util.c++:393: error: e = kj/async-io-unix.c++:1523: failed: connect(): Can't assign requested address; address = 127.0.0.1:54089
stack: 1047ed1b3 1047eaf6f 1047eb5ac 102230be0 102260263 102ef0638 1047e2bb4 1047cc110 10330f003 10330f9a3 1033118c7 1032f5448 1032fb018 10330c32c 1021f4230 102cd883c 102bc8ff4; sentryErrorContext = jsgInternalError; wdErrId = ph3npsvuota0svnreqrr7dkf
workerd/io/worker.c++:2168: info: uncaught exception; source = Uncaught (in promise); stack = Error: internal error; reference = ph3npsvuota0svnreqrr7dkf
workerd/io/io-context.c++:389: info: uncaught exception; exception = workerd/jsg/_virtual_includes/iterator/workerd/jsg/value.h:1532: failed: jsg.Error: internal error; reference = ph3npsvuota0svnreqrr7dkf
stack: 102c95a48 102c96613 1047bf3db 102acad8c 102acb178
Error running worker: Error: internal error; reference = ph3npsvuota0svnreqrr7dkf
at WorkersSnapshotEnvironment.readSnapshotFile (Users/dmaretskyi/Code/dxos/vpw-repro/node_modules/.pnpm/@[email protected]_@[email protected]_@[email protected][email protected]/node_modules/@cloudflare/vitest-pool-workers/dist/worker/lib/cloudflare/test-runner.mjs:45:17)
at SnapshotState.create (Users/dmaretskyi/Code/dxos/vpw-repro/node_modules/.pnpm/@[email protected]/node_modules/@vitest/snapshot/dist/index.js:1926:19)
at SnapshotClient.setup (Users/dmaretskyi/Code/dxos/vpw-repro/node_modules/.pnpm/@[email protected]/node_modules/@vitest/snapshot/dist/index.js:2220:39)
at WorkersTestRunner.onBeforeRunSuite (Users/dmaretskyi/Code/dxos/vpw-repro/node_modules/.pnpm/[email protected]/node_modules/vitest/dist/runners.js:182:53)
at runSuite (Users/dmaretskyi/Code/dxos/vpw-repro/node_modules/.pnpm/@[email protected]/node_modules/@vitest/runner/dist/chunk-hooks.js:1683:2)
at runFiles (Users/dmaretskyi/Code/dxos/vpw-repro/node_modules/.pnpm/@[email protected]/node_modules/@vitest/runner/dist/chunk-hooks.js:1787:3)
at startTests (Users/dmaretskyi/Code/dxos/vpw-repro/node_modules/.pnpm/@[email protected]/node_modules/@vitest/runner/dist/chunk-hooks.js:1820:3)
at Users/dmaretskyi/Code/dxos/vpw-repro/node_modules/.pnpm/[email protected]/node_modules/vitest/dist/chunks/runBaseTests.9Ij9_de-.js:117:26
at withEnv (Users/dmaretskyi/Code/dxos/vpw-repro/node_modules/.pnpm/[email protected]/node_modules/vitest/dist/chunks/runBaseTests.9Ij9_de-.js:84:3)
at run (Users/dmaretskyi/Code/dxos/vpw-repro/node_modules/.pnpm/[email protected]/node_modules/vitest/dist/chunks/runBaseTests.9Ij9_de-.js:109:2)
cc @emily-shen and @lrapoport-cf from the original issue
I found related issue https://github.com/cloudflare/workers-sdk/issues/7569, this error is not reproducable when the flag poolOptions.workers.singleWorker is enabled
Thanks for the great reproduction! I can see this happening locally too. Adding to the backlog to be looked into.