forklift icon indicating copy to clipboard operation
forklift copied to clipboard
verified publisher icon paritytech

Connection errors lead to build failures?

Open bkchr opened this issue 2 months ago • 2 comments 

    Using forklift, storage: gcs, compressor: zstd-3, uploader threads: 4
    Unable to write data to bucket "parity-ci-forklift-regional", file "substrate_test_runtime_ba29da5e9200646c7959ee09ce6d2f22d5c13fc1_zstd-3": Compressor error: io.copy error, inner error: stat target/forklift/substrate_test_runtime_ba29da5e9200646c7959ee09ce6d2f22d5c13fc1-stderr: no such file or directory
    upload error: Compressor error: io.copy error, inner error: stat target/forklift/substrate_test_runtime_ba29da5e9200646c7959ee09ce6d2f22d5c13fc1-stderr: no such file or directory
    Unable to write data to bucket "parity-ci-forklift-regional", file "substrate_test_runtime_ba29da5e9200646c7959ee09ce6d2f22d5c13fc1_zstd-3": Compressor error: io.copy error, inner error: stat target/forklift/substrate_test_runtime_ba29da5e9200646c7959ee09ce6d2f22d5c13fc1-stderr: no such file or directory
    upload error: Compressor error: io.copy error, inner error: stat target/forklift/substrate_test_runtime_ba29da5e9200646c7959ee09ce6d2f22d5c13fc1-stderr: no such file or directory
    Unable to write data to bucket "parity-ci-forklift-regional", file "substrate_test_runtime_ba29da5e9200646c7959ee09ce6d2f22d5c13fc1_zstd-3": Compressor error: io.copy error, inner error: stat target/forklift/substrate_test_runtime_ba29da5e9200646c7959ee09ce6d2f22d5c13fc1-stderr: no such file or directory
    upload error: Compressor error: io.copy error, inner error: stat target/forklift/substrate_test_runtime_ba29da5e9200646c7959ee09ce6d2f22d5c13fc1-stderr: no such file or directory
    Failed to upload artifact for 'substrate_test_runtime_ba29da5e9200646c7959ee09ce6d2f22d5c13fc1, 9030cc672c8dcd97'
    Unable to write data to bucket "parity-ci-forklift-regional", file "substrate_test_runtime_b92b71932f501f557cf5de5d4d17a48deaa223c8_zstd-3": Compressor error: io.copy error, inner error: stat target/forklift/substrate_test_runtime_b92b71932f501f557cf5de5d4d17a48deaa223c8-stderr: no such file or directory
    upload error: Compressor error: io.copy error, inner error: stat target/forklift/substrate_test_runtime_b92b71932f501f557cf5de5d4d17a48deaa223c8-stderr: no such file or directory
    Unable to write data to bucket "parity-ci-forklift-regional", file "substrate_test_runtime_b92b71932f501f557cf5de5d4d17a48deaa223c8_zstd-3": Compressor error: io.copy error, inner error: stat target/forklift/substrate_test_runtime_b92b71932f501f557cf5de5d4d17a48deaa223c8-stderr: no such file or directory
    upload error: Compressor error: io.copy error, inner error: stat target/forklift/substrate_test_runtime_b92b71932f501f557cf5de5d4d17a48deaa223c8-stderr: no such file or directory
    Unable to write data to bucket "parity-ci-forklift-regional", file "substrate_test_runtime_b92b71932f501f557cf5de5d4d17a48deaa223c8_zstd-3": Compressor error: io.copy error, inner error: stat target/forklift/substrate_test_runtime_b92b71932f501f557cf5de5d4d17a48deaa223c8-stderr: no such file or directory
    upload error: Compressor error: io.copy error, inner error: stat target/forklift/substrate_test_runtime_b92b71932f501f557cf5de5d4d17a48deaa223c8-stderr: no such file or directory
    Failed to upload artifact for 'substrate_test_runtime_b92b71932f501f557cf5de5d4d17a48deaa223c8, 99724a9a5314ab20'

See: https://github.com/paritytech/polkadot-sdk/actions/runs/19166121427/job/54787276817

I have seen this already multiple times that we see these connection errors by forklift and then the build fails in mysterious ways. Can not say for 100% that they are related.

bkchr avatar Nov 07 '25 11:11 bkchr

It not a connection issue, it's uploader thread could not find the file for cache artifact. Wrapper runs rustc command as is and report the result to the uploader, which attempts to pack and upload artifacts in the background. For some reason there is no file with stderr rustc output, which should be prepared by the wrapper. This is a known problem, and I am still investigating it. However, this error is triggered often (if not always), even for successful runs, and always for substrate_test_runtime, for example:

  • https://github.com/paritytech/polkadot-sdk/actions/runs/19158246656/job/54763563682
  • https://github.com/paritytech/polkadot-sdk/actions/runs/19167584729/job/54791988483
  • https://github.com/paritytech/polkadot-sdk/actions/runs/19170572304/job/54801901513

In your example (https://github.com/paritytech/polkadot-sdk/actions/runs/19166121427/job/54787276817) tests run with --cargo-quiet so there are no intermediate build logs, but with timestamps there are almost 10 minutes between upload error and failed build

Fri, 07 Nov 2025 11:02:26 GMT upload error: Compressor error: io.copy error, inner error: stat target/forklift/substrate_test_runtime_b92b71932f501f557cf5de5d4d17a48deaa223c8-stderr: no such file or directory
Fri, 07 Nov 2025 11:02:26 GMT Unable to write data to bucket "parity-ci-forklift-regional", file "substrate_test_runtime_b92b71932f501f557cf5de5d4d17a48deaa223c8_zstd-3": Compressor error: io.copy error, inner error: stat target/forklift/substrate_test_runtime_b92b71932f501f557cf5de5d4d17a48deaa223c8-stderr: no such file or directory
Fri, 07 Nov 2025 11:02:26 GMT upload error: Compressor error: io.copy error, inner error: stat target/forklift/substrate_test_runtime_b92b71932f501f557cf5de5d4d17a48deaa223c8-stderr: no such file or directory
Fri, 07 Nov 2025 11:02:26 GMT Failed to upload artifact for 'substrate_test_runtime_b92b71932f501f557cf5de5d4d17a48deaa223c8, 99724a9a5314ab20'
Fri, 07 Nov 2025 11:07:12 GMT ⚡ Found 4 strongly connected components which includes at least one cycle each
.......
Fri, 07 Nov 2025 11:13:58 GMT error: failed to run custom build command for `revive-dev-runtime v0.1.0 (/__w/polkadot-sdk/polkadot-sdk/substrate/frame/revive/dev-node/runtime)`
Fri, 07 Nov 2025 11:13:58 GMT note: To improve backtraces for build dependencies, set the CARGO_PROFILE_RELEASE_BUILD_OVERRIDE_DEBUG=true environment variable to enable debug information generation.
Fri, 07 Nov 2025 11:13:58 GMT
Fri, 07 Nov 2025 11:13:58 GMT Caused by:
Fri, 07 Nov 2025 11:13:58 GMT process didn't exit successfully: `/__w/polkadot-sdk/polkadot-sdk/target/release/build/revive-dev-runtime-cb69ca692a31fde6/build-script-build` (exit status: 101)
Fri, 07 Nov 2025 11:13:58 GMT --- stdout

I don't think they are directly related. But maybe they both are symptoms of some sort of a race condition with nested build (missing forklift file and parse exception: Reached function end without seeing End opcode)

AndWeHaveAPlan avatar Nov 07 '25 15:11 AndWeHaveAPlan

https://github.com/paritytech/polkadot-sdk/actions/runs/19260653265/job/55064671170 is another one I have seen today.

bkchr avatar Nov 11 '25 10:11 bkchr