rust
rust copied to clipboard
rust-lang
High failure rate on Windows MSVC CI with filesystem errors
Starting around 2024-06-27, the rust-lang CI has started to encounter a very high failure rate on the MSVC Windows builders (~15% of all builds?). These builders are encountering various filesystems errors, such as "Access is denied", "used by another process", "cannot open file", etc.
The following is a sample of the errors seen:
2024-05-15T17:13:48 45_auto - x86_64-msvc.txt "../logs-rust-lang-ci-rust\\logs_9099930545.zip" found match https://github.com/rust-lang-ci/rust/actions/runs/9099930545
2024-05-15T18:00:27.5922941Z --- stdout -------------------------------
2024-05-15T18:00:27.5924398Z MONO_ITEM fn inline::inlined_function @@ local_inlining.8ec87a7fbea559a-user2[Internal] local_inlining.8ec87a7fbea559a-user1[Internal]
2024-05-15T18:00:27.5925974Z MONO_ITEM fn non_user::baz @@ local_inlining.8ec87a7fbea559a-non_user[External]
2024-05-15T18:00:27.5926735Z MONO_ITEM fn user1::foo @@ local_inlining.8ec87a7fbea559a-user1[External]
2024-05-15T18:00:27.5927446Z MONO_ITEM fn user2::bar @@ local_inlining.8ec87a7fbea559a-user2[External]
2024-05-15T18:00:27.5927993Z ------------------------------------------
2024-05-15T18:00:27.5928393Z --- stderr -------------------------------
2024-05-15T18:00:27.5929309Z error: couldn't create a temp dir: Access is denied. (os error 5) at path "C:\\a\\_temp\\msys64\\tmp\\rustcFKhJYE"
2024-05-15T18:00:27.5929887Z
2024-05-15T18:00:27.5930041Z error: aborting due to 1 previous error
2024-05-15T18:00:27.5930467Z ------------------------------------------
2024-05-15T18:00:27.5930772Z
2024-05-15T18:00:27.5930778Z
2024-05-15T18:00:27.5930784Z
2024-05-15T18:00:27.5930879Z failures:
2024-05-15T18:00:27.5931329Z [codegen-units] tests\codegen-units\partitioning\local-inlining.rs
2024-06-21T23:51:42 45_auto - x86_64-msvc.txt "../logs-rust-lang-ci-rust\\logs_9621092085.zip" found match https://github.com/rust-lang-ci/rust/actions/runs/9621092085
2024-06-22T00:38:19.3827973Z --- stderr -------------------------------
2024-06-22T00:38:19.3828801Z error: couldn't create a temp dir: Access is denied. (os error 5) at path "C:\\a\\_temp\\msys64\\tmp\\rustcMwb8Le"
2024-06-22T00:38:19.3829381Z
2024-06-22T00:38:19.3829535Z error: aborting due to 1 previous error
2024-06-22T00:38:19.3829971Z ------------------------------------------
2024-06-22T00:38:19.3830211Z
2024-06-22T00:38:19.3830217Z
2024-06-22T00:38:19.3830223Z
2024-06-22T00:38:19.3830318Z failures:
2024-06-22T00:38:19.3830672Z [ui] tests\ui\lto\lto-duplicate-symbols.rs
2024-06-27T21:05:13 48_auto - x86_64-msvc-ext.txt "../logs-rust-lang-ci-rust\\logs_9703582059.zip" found match https://github.com/rust-lang-ci/rust/actions/runs/9703582059
2024-06-27T22:37:13.7595098Z [TIMING] core::build_steps::tool::LlvmBitcodeLinker { compiler: Compiler { stage: 0, host: x86_64-pc-windows-msvc }, target: x86_64-pc-windows-msvc, extra_features: [] } -- 0.396
2024-06-27T22:37:13.7611176Z thread 'main' panicked at src/lib.rs:1713:17:
2024-06-27T22:37:13.7613255Z failed to copy `C:\a\rust\rust\build\x86_64-pc-windows-msvc\stage0-rustc\x86_64-pc-windows-msvc\release\rustc-main.exe` to `C:\a\rust\rust\build\x86_64-pc-windows-msvc\stage1\bin\rustc.exe`: The process cannot access the file because it is being used by another process. (os error 32)
2024-06-27T23:42:02 48_auto - x86_64-msvc-ext.txt "../logs-rust-lang-ci-rust\\logs_9705061663.zip" found match https://github.com/rust-lang-ci/rust/actions/runs/9705061663
2024-06-28T01:08:22.3300116Z Finished `release` profile [optimized] target(s) in 0.41s
2024-06-28T01:08:22.3517842Z thread 'main' panicked at src/lib.rs:1713:17:
2024-06-28T01:08:22.3520184Z failed to copy `C:\a\rust\rust\build\x86_64-pc-windows-msvc\stage1-tools\x86_64-pc-windows-msvc\release\miri.exe` to `C:\a\rust\rust\build\x86_64-pc-windows-msvc\stage1-tools-bin\miri.exe`: The process cannot access the file because it is being used by another process. (os error 32)
2024-06-28T01:10:46 48_auto - x86_64-msvc-ext.txt "../logs-rust-lang-ci-rust\\logs_9705841544.zip" found match https://github.com/rust-lang-ci/rust/actions/runs/9705841544
2024-06-28T02:37:41.9366820Z Finished `release` profile [optimized] target(s) in 0.39s
2024-06-28T02:37:41.9583934Z thread 'main' panicked at src/lib.rs:1713:17:
2024-06-28T02:37:41.9586316Z failed to copy `C:\a\rust\rust\build\x86_64-pc-windows-msvc\stage1-tools\x86_64-pc-windows-msvc\release\miri.exe` to `C:\a\rust\rust\build\x86_64-pc-windows-msvc\stage1-tools-bin\miri.exe`: The process cannot access the file because it is being used by another process. (os error 32)
2024-06-28T06:30:49 51_auto - dist-x86_64-msvc.txt "../logs-rust-lang-ci-rust\\logs_9708494009.zip" found match https://github.com/rust-lang-ci/rust/actions/runs/9708494009
2024-06-28T07:23:27.6412089Z = note: LINK : fatal error LNK1104: cannot open file 'C:\a\rust\rust\build\x86_64-pc-windows-msvc\stage1-rustc\x86_64-pc-windows-msvc\release\deps\rustc_main-2a5d00462f8c666b.exe'
2024-06-28T07:23:27.6413223Z
2024-06-28T07:23:27.6413471Z
2024-06-28T07:23:27.6413651Z
2024-06-28T07:23:27.6558489Z error: could not compile `rustc-main` (bin "rustc-main") due to 1 previous error
2024-06-28T09:36:50 48_auto - x86_64-msvc-ext.txt "../logs-rust-lang-ci-rust\\logs_9710725566.zip" found match https://github.com/rust-lang-ci/rust/actions/runs/9710725566
2024-06-28T11:03:13.0991016Z thread 'main' panicked at src/lib.rs:1712:17:
2024-06-28T11:03:13.0993328Z failed to copy `C:\a\rust\rust\build\x86_64-pc-windows-msvc\stage1-tools\x86_64-pc-windows-msvc\release\miri.exe` to `C:\a\rust\rust\build\x86_64-pc-windows-msvc\stage1-tools-bin\miri.exe`: The process cannot access the file because it is being used by another process. (os error 32)
2024-06-28T11:05:47 48_auto - x86_64-msvc-ext.txt "../logs-rust-lang-ci-rust\\logs_9711788661.zip" found match https://github.com/rust-lang-ci/rust/actions/runs/9711788661
2024-06-28T12:32:22.9420083Z Finished `release` profile [optimized] target(s) in 0.39s
2024-06-28T12:32:22.9679265Z thread 'main' panicked at src/lib.rs:1712:17:
2024-06-28T12:32:22.9681613Z failed to copy `C:\a\rust\rust\build\x86_64-pc-windows-msvc\stage1-tools\x86_64-pc-windows-msvc\release\miri.exe` to `C:\a\rust\rust\build\x86_64-pc-windows-msvc\stage1-tools-bin\miri.exe`: The process cannot access the file because it is being used by another process. (os error 32)
2024-06-28T14:28:38 48_auto - x86_64-msvc-ext.txt "../logs-rust-lang-ci-rust\\logs_9714425374.zip" found match https://github.com/rust-lang-ci/rust/actions/runs/9714425374
2024-06-28T16:03:54.7079854Z Finished `release` profile [optimized] target(s) in 0.38s
2024-06-28T16:03:54.7298453Z thread 'main' panicked at src/lib.rs:1712:17:
2024-06-28T16:03:54.7300791Z failed to copy `C:\a\rust\rust\build\x86_64-pc-windows-msvc\stage1-tools\x86_64-pc-windows-msvc\release\miri.exe` to `C:\a\rust\rust\build\x86_64-pc-windows-msvc\stage1-tools-bin\miri.exe`: The process cannot access the file because it is being used by another process. (os error 32)
2024-06-28T16:06:30 48_auto - x86_64-msvc-ext.txt "../logs-rust-lang-ci-rust\\logs_9715662476.zip" found match https://github.com/rust-lang-ci/rust/actions/runs/9715662476
2024-06-28T17:40:10.3608665Z Finished `release` profile [optimized] target(s) in 0.39s
2024-06-28T17:40:10.3838225Z thread 'main' panicked at src/lib.rs:1712:17:
2024-06-28T17:40:10.3840547Z failed to copy `C:\a\rust\rust\build\x86_64-pc-windows-msvc\stage1-tools\x86_64-pc-windows-msvc\release\miri.exe` to `C:\a\rust\rust\build\x86_64-pc-windows-msvc\stage1-tools-bin\miri.exe`: The process cannot access the file because it is being used by another process. (os error 32)
2024-06-28T22:09:33 48_auto - x86_64-msvc-ext.txt "../logs-rust-lang-ci-rust\\logs_9719051471.zip" found match https://github.com/rust-lang-ci/rust/actions/runs/9719051471
2024-06-28T23:42:05.4732924Z Finished `release` profile [optimized] target(s) in 0.40s
2024-06-28T23:42:05.4950520Z thread 'main' panicked at src/lib.rs:1712:17:
2024-06-28T23:42:05.4952824Z failed to copy `C:\a\rust\rust\build\x86_64-pc-windows-msvc\stage1-tools\x86_64-pc-windows-msvc\release\miri.exe` to `C:\a\rust\rust\build\x86_64-pc-windows-msvc\stage1-tools-bin\miri.exe`: The process cannot access the file because it is being used by another process. (os error 32)
2024-06-29T02:00:21 48_auto - x86_64-msvc-ext.txt "../logs-rust-lang-ci-rust\\logs_9720709773.zip" found match https://github.com/rust-lang-ci/rust/actions/runs/9720709773
2024-06-29T03:30:57.6850989Z Finished `release` profile [optimized] target(s) in 0.39s
2024-06-29T03:30:57.7066422Z thread 'main' panicked at src/lib.rs:1712:17:
2024-06-29T03:30:57.7068713Z failed to copy `C:\a\rust\rust\build\x86_64-pc-windows-msvc\stage1-tools\x86_64-pc-windows-msvc\release\miri.exe` to `C:\a\rust\rust\build\x86_64-pc-windows-msvc\stage1-tools-bin\miri.exe`: The process cannot access the file because it is being used by another process. (os error 32)
2024-06-29T03:33:22 48_auto - x86_64-msvc-ext.txt "../logs-rust-lang-ci-rust\\logs_9721279263.zip" found match https://github.com/rust-lang-ci/rust/actions/runs/9721279263
2024-06-29T04:58:15.8326565Z Finished `release` profile [optimized] target(s) in 0.38s
2024-06-29T04:58:15.8683528Z thread 'main' panicked at src/lib.rs:1712:17:
2024-06-29T04:58:15.8685846Z failed to copy `C:\a\rust\rust\build\x86_64-pc-windows-msvc\stage1-tools\x86_64-pc-windows-msvc\release\miri.exe` to `C:\a\rust\rust\build\x86_64-pc-windows-msvc\stage1-tools-bin\miri.exe`: The process cannot access the file because it is being used by another process. (os error 32)
2024-06-29T16:14:43 48_auto - x86_64-msvc-ext.txt "../logs-rust-lang-ci-rust\\logs_9725396338.zip" found match https://github.com/rust-lang-ci/rust/actions/runs/9725396338
2024-06-29T17:48:11.1950874Z Finished `release` profile [optimized] target(s) in 0.39s
2024-06-29T17:48:11.2163942Z thread 'main' panicked at src/lib.rs:1697:17:
2024-06-29T17:48:11.2166059Z failed to copy `C:\a\rust\rust\build\x86_64-pc-windows-msvc\stage1-tools\x86_64-pc-windows-msvc\release\miri.exe` to `C:\a\rust\rust\build\x86_64-pc-windows-msvc\stage1-tools-bin\miri.exe`: The process cannot access the file because it is being used by another process. (os error 32)
2024-06-30T05:53:23 48_auto - x86_64-msvc-ext.txt "../logs-rust-lang-ci-rust\\logs_9729777789.zip" found match https://github.com/rust-lang-ci/rust/actions/runs/9729777789
2024-06-30T07:18:43.4197375Z Finished `release` profile [optimized] target(s) in 0.39s
2024-06-30T07:18:43.4410518Z thread 'main' panicked at src/lib.rs:1697:17:
2024-06-30T07:18:43.4412624Z failed to copy `C:\a\rust\rust\build\x86_64-pc-windows-msvc\stage1-tools\x86_64-pc-windows-msvc\release\miri.exe` to `C:\a\rust\rust\build\x86_64-pc-windows-msvc\stage1-tools-bin\miri.exe`: The process cannot access the file because it is being used by another process. (os error 32)
2024-06-30T08:15:48 48_auto - x86_64-msvc-ext.txt "../logs-rust-lang-ci-rust\\logs_9730548394.zip" found match https://github.com/rust-lang-ci/rust/actions/runs/9730548394
2024-06-30T09:40:47.0630932Z thread 'main' panicked at src/lib.rs:1697:17:
2024-06-30T09:40:47.0633012Z failed to copy `C:\a\rust\rust\build\x86_64-pc-windows-msvc\stage1-tools\x86_64-pc-windows-msvc\release\miri.exe` to `C:\a\rust\rust\build\x86_64-pc-windows-msvc\stage1-tools-bin\miri.exe`: The process cannot access the file because it is being used by another process. (os error 32)
2024-06-30T09:43:08 49_auto - x86_64-msvc-ext.txt "../logs-rust-lang-ci-rust\\logs_9731047889.zip" found match https://github.com/rust-lang-ci/rust/actions/runs/9731047889
2024-06-30T11:07:56.3788705Z Finished `release` profile [optimized] target(s) in 0.39s
2024-06-30T11:07:56.4006304Z thread 'main' panicked at src/lib.rs:1697:17:
2024-06-30T11:07:56.4008397Z failed to copy `C:\a\rust\rust\build\x86_64-pc-windows-msvc\stage1-tools\x86_64-pc-windows-msvc\release\miri.exe` to `C:\a\rust\rust\build\x86_64-pc-windows-msvc\stage1-tools-bin\miri.exe`: The process cannot access the file because it is being used by another process. (os error 32)
2024-06-30T15:35:38 57_auto - dist-x86_64-msvc-alt.txt "../logs-rust-lang-ci-rust\\logs_9733131162.zip" found match https://github.com/rust-lang-ci/rust/actions/runs/9733131162
2024-06-30T16:21:12.2870212Z thread 'main' panicked at src/lib.rs:1697:17:
2024-06-30T16:21:12.2871936Z failed to copy `C:\a\rust\rust\build\x86_64-pc-windows-msvc\stage2\bin\rustdoc.exe` to `C:\a\rust\rust\build\tmp\tarball\rustc\x86_64-pc-windows-msvc\image\bin\rustdoc.exe`: The process cannot access the file because it is being used by another process. (os error 32)
2024-06-30T18:43:09 49_auto - x86_64-msvc-ext.txt "../logs-rust-lang-ci-rust\\logs_9734174198.zip" found match https://github.com/rust-lang-ci/rust/actions/runs/9734174198
2024-06-30T20:12:39.4763461Z thread 'main' panicked at src/lib.rs:1668:17:
2024-06-30T20:12:39.4765689Z failed to copy `C:\a\rust\rust\build\x86_64-pc-windows-msvc\stage1-tools\x86_64-pc-windows-msvc\release\miri.exe` to `C:\a\rust\rust\build\x86_64-pc-windows-msvc\stage1-tools-bin\miri.exe`: The process cannot access the file because it is being used by another process. (os error 32)
2024-07-01T11:52:50 49_auto - x86_64-msvc-ext.txt "../logs-rust-lang-ci-rust\\logs_9743687544.zip" found match https://github.com/rust-lang-ci/rust/actions/runs/9743687544
2024-07-01T13:17:07.6056787Z thread 'main' panicked at src/lib.rs:1668:17:
2024-07-01T13:17:07.6059053Z failed to copy `C:\a\rust\rust\build\x86_64-pc-windows-msvc\stage1-tools\x86_64-pc-windows-msvc\release\miri.exe` to `C:\a\rust\rust\build\x86_64-pc-windows-msvc\stage1-tools-bin\miri.exe`: The process cannot access the file because it is being used by another process. (os error 32)
2024-07-01T13:19:31 52_auto - dist-x86_64-msvc.txt "../logs-rust-lang-ci-rust\\logs_9745004151.zip" found match https://github.com/rust-lang-ci/rust/actions/runs/9745004151
2024-07-01T14:07:36.3302291Z = note: LINK : fatal error LNK1104: cannot open file 'C:\a\rust\rust\build\x86_64-pc-windows-msvc\stage1-rustc\x86_64-pc-windows-msvc\release\deps\rustc_main-8c9e059926f63509.exe'
2024-07-01T14:07:36.3303400Z
2024-07-01T14:07:36.3303642Z
2024-07-01T14:07:36.3303843Z
2024-07-01T14:07:36.3425523Z error: could not compile `rustc-main` (bin "rustc-main") due to 1 previous error
2024-07-01T14:07:36.3853896Z [2024-07-01T14:07:36.385Z INFO opt_dist::timer] Section `Stage 1 (Rustc PGO) > Build PGO optimized rustc` ended: FAIL (562.38s)`
2024-07-01T14:07:36.3855267Z [2024-07-01T14:07:36.385Z INFO opt_dist::timer] Section `Stage 1 (Rustc PGO)` ended: FAIL (2419.29s)`
2024-07-01T14:07:36.3856022Z [2024-07-01T14:07:36.385Z INFO opt_dist] Timer results
2024-07-01T14:07:36.3856571Z -----------------------------------------------------------------
2024-07-01T16:22:23 49_auto - x86_64-msvc-ext.txt "../logs-rust-lang-ci-rust\\logs_9747722395.zip" found match https://github.com/rust-lang-ci/rust/actions/runs/9747722395
2024-07-01T17:54:39.4755072Z thread 'main' panicked at src/lib.rs:1668:17:
2024-07-01T17:54:39.4757239Z failed to copy `C:\a\rust\rust\build\x86_64-pc-windows-msvc\stage1-tools\x86_64-pc-windows-msvc\release\miri.exe` to `C:\a\rust\rust\build\x86_64-pc-windows-msvc\stage1-tools-bin\miri.exe`: The process cannot access the file because it is being used by another process. (os error 32)
2024-07-01T17:56:59 49_auto - x86_64-msvc-ext.txt "../logs-rust-lang-ci-rust\\logs_9748873919.zip" found match https://github.com/rust-lang-ci/rust/actions/runs/9748873919
2024-07-01T19:31:13.5443614Z thread 'main' panicked at src/lib.rs:1668:17:
2024-07-01T19:31:13.5445798Z failed to copy `C:\a\rust\rust\build\x86_64-pc-windows-msvc\stage1-tools\x86_64-pc-windows-msvc\release\miri.exe` to `C:\a\rust\rust\build\x86_64-pc-windows-msvc\stage1-tools-bin\miri.exe`: The process cannot access the file because it is being used by another process. (os error 32)
2024-07-02T00:13:28 57_auto - dist-x86_64-msvc-alt.txt "../logs-rust-lang-ci-rust\\logs_9752796989.zip" found match https://github.com/rust-lang-ci/rust/actions/runs/9752796989
2024-07-02T01:01:09.3044765Z thread 'main' panicked at src/lib.rs:1668:17:
2024-07-02T01:01:09.3046561Z failed to copy `C:\a\rust\rust\build\x86_64-pc-windows-msvc\stage2\bin\rustdoc.exe` to `C:\a\rust\rust\build\tmp\tarball\rustc\x86_64-pc-windows-msvc\image\bin\rustdoc.exe`: The process cannot access the file because it is being used by another process. (os error 32)
2024-07-02T01:03:31 49_auto - x86_64-msvc-ext.txt "../logs-rust-lang-ci-rust\\logs_9753269047.zip" found match https://github.com/rust-lang-ci/rust/actions/runs/9753269047
2024-07-02T02:30:23.7862414Z thread 'main' panicked at src/lib.rs:1668:17:
2024-07-02T02:30:23.7864496Z failed to copy `C:\a\rust\rust\build\x86_64-pc-windows-msvc\stage1-tools\x86_64-pc-windows-msvc\release\miri.exe` to `C:\a\rust\rust\build\x86_64-pc-windows-msvc\stage1-tools-bin\miri.exe`: The process cannot access the file because it is being used by another process. (os error 32)
2024-07-02T12:53:47 52_auto - dist-x86_64-msvc.txt "../logs-rust-lang-ci-rust\\logs_9761329541.zip" found match https://github.com/rust-lang-ci/rust/actions/runs/9761329541
2024-07-02T13:48:36.1493324Z = note: LINK : fatal error LNK1104: cannot open file 'C:\a\rust\rust\build\x86_64-pc-windows-msvc\stage1-rustc\x86_64-pc-windows-msvc\release\deps\rustc_main-8c9e059926f63509.exe'
2024-07-02T13:48:36.1494467Z
2024-07-02T13:48:36.1494734Z
2024-07-02T13:48:36.1494906Z
2024-07-02T13:48:36.1608837Z error: could not compile `rustc-main` (bin "rustc-main") due to 1 previous error
2024-07-02T13:51:08 49_auto - x86_64-msvc-ext.txt "../logs-rust-lang-ci-rust\\logs_9762243104.zip" found match https://github.com/rust-lang-ci/rust/actions/runs/9762243104
2024-07-02T15:17:39.8880874Z Finished `release` profile [optimized] target(s) in 0.41s
2024-07-02T15:17:39.9100976Z thread 'main' panicked at src/lib.rs:1668:17:
2024-07-02T15:17:39.9103158Z failed to copy `C:\a\rust\rust\build\x86_64-pc-windows-msvc\stage1-tools\x86_64-pc-windows-msvc\release\miri.exe` to `C:\a\rust\rust\build\x86_64-pc-windows-msvc\stage1-tools-bin\miri.exe`: The process cannot access the file because it is being used by another process. (os error 32)
2024-07-04T13:14:14 49_auto - x86_64-msvc-ext.txt "../logs-rust-lang-ci-rust\\logs_9795018238.zip" found match https://github.com/rust-lang-ci/rust/actions/runs/9795018238
2024-07-04T14:21:22.8765500Z
2024-07-04T14:21:35.0891504Z error: failed to remove file `C:\a\rust\rust\build\x86_64-pc-windows-msvc\stage2-tools\x86_64-pc-windows-msvc\release\cargo.exe`
2024-07-04T14:21:35.0892319Z
2024-07-04T14:21:35.0892437Z Caused by:
2024-07-04T14:21:35.0892781Z Access is denied. (os error 5)
2024-07-06T09:36:09 49_auto - x86_64-msvc-ext.txt "../logs-rust-lang-ci-rust\\logs_9818299526.zip" found match https://github.com/rust-lang-ci/rust/actions/runs/9818299526
2024-07-06T10:39:24.5995051Z
2024-07-06T10:39:38.3349340Z error: failed to remove file `C:\a\rust\rust\build\x86_64-pc-windows-msvc\stage2-tools\x86_64-pc-windows-msvc\release\cargo.exe`
2024-07-06T10:39:38.3350150Z
2024-07-06T10:39:38.3350315Z Caused by:
2024-07-06T10:39:38.3350655Z Access is denied. (os error 5)
2024-07-06T10:55:59 45_auto - x86_64-msvc-ext.txt "../logs-rust-lang-ci-rust\\logs_9818726134.zip" found match https://github.com/rust-lang-ci/rust/actions/runs/9818726134
2024-07-06T12:05:59.8544699Z = note: LINK : fatal error LNK1104: cannot open file 'C:\a\rust\rust\build\x86_64-pc-windows-msvc\stage1-tools\x86_64-pc-windows-msvc\release\deps\rustdoc_tool_binary-a7c1152b8abe3e78.exe'
2024-07-06T12:05:59.8545901Z
2024-07-06T12:05:59.8546164Z
2024-07-06T12:05:59.8546344Z
2024-07-06T12:05:59.8624055Z error: could not compile `rustdoc-tool` (bin "rustdoc_tool_binary") due to 1 previous error
2024-07-06T14:21:23 45_auto - x86_64-msvc-ext.txt "../logs-rust-lang-ci-rust\\logs_9819935411.zip" found match https://github.com/rust-lang-ci/rust/actions/runs/9819935411
2024-07-06T15:54:19.8152269Z = note: LINK : fatal error LNK1104: cannot open file 'C:\a\rust\rust\build\x86_64-pc-windows-msvc\stage1-tools\x86_64-pc-windows-msvc\release\deps\miri-ff57ced5dff426fc.exe'
2024-07-06T15:54:19.8153375Z
2024-07-06T15:54:19.8153630Z
2024-07-06T15:54:19.8153804Z
2024-07-06T15:54:19.8338918Z error: could not compile `miri` (bin "miri") due to 1 previous error
2024-07-07T10:30:30 47_auto - x86_64-msvc.txt "../logs-rust-lang-ci-rust\\logs_9826600375.zip" found match https://github.com/rust-lang-ci/rust/actions/runs/9826600375
2024-07-07T11:29:01.8968721Z error: failed to remove file `C:\a\rust\rust\build\x86_64-pc-windows-msvc\stage1-std\x86_64-pc-windows-msvc\release\deps\libcompiler_builtins-86bc7a5c308aedf8.rlib`
2024-07-07T11:29:01.8970410Z
2024-07-07T11:29:01.8970609Z Caused by:
2024-07-07T11:29:01.8971101Z Access is denied. (os error 5)
2024-07-07T18:50:57 48_auto - i686-msvc.txt "../logs-rust-lang-ci-rust\\logs_9829572529.zip" found match https://github.com/rust-lang-ci/rust/actions/runs/9829572529
2024-07-07T19:50:51.1512300Z error: failed to remove file `C:\a\rust\rust\build\i686-pc-windows-msvc\stage1-std\i686-pc-windows-msvc\release\deps\libcompiler_builtins-7b6e8fab8041690f.rlib`
2024-07-07T19:50:51.1513505Z
2024-07-07T19:50:51.1513617Z Caused by:
2024-07-07T19:50:51.1513898Z Access is denied. (os error 5)
2024-07-07T19:52:17.1913925Z make: *** [Makefile:102: ci-msvc-ps1] Error 1
2024-07-07T22:06:12 47_auto - x86_64-msvc.txt "../logs-rust-lang-ci-rust\\logs_9830624681.zip" found match https://github.com/rust-lang-ci/rust/actions/runs/9830624681
2024-07-07T23:00:35.2960420Z error: failed to remove file `C:\a\rust\rust\build\x86_64-pc-windows-msvc\stage1-std\x86_64-pc-windows-msvc\release\deps\libcompiler_builtins-86bc7a5c308aedf8.rlib`
2024-07-07T23:00:35.2962043Z
2024-07-07T23:00:35.2962208Z Caused by:
2024-07-07T23:00:35.2962633Z Access is denied. (os error 5)
2024-07-08T09:07:11 47_auto - x86_64-msvc.txt "../logs-rust-lang-ci-rust\\logs_9836726350.zip" found match https://github.com/rust-lang-ci/rust/actions/runs/9836726350
2024-07-08T10:02:01.6196210Z error: failed to remove file `C:\a\rust\rust\build\x86_64-pc-windows-msvc\stage1-std\x86_64-pc-windows-msvc\release\deps\libcompiler_builtins-86bc7a5c308aedf8.rlib`
2024-07-08T10:02:01.6198226Z
2024-07-08T10:02:01.6198438Z Caused by:
2024-07-08T10:02:01.6198942Z Access is denied. (os error 5)
2024-07-11T05:27:59 52_auto - dist-x86_64-msvc.txt "../logs-rust-lang-ci-rust\\logs_9886006325.zip" found match https://github.com/rust-lang-ci/rust/actions/runs/9886006325
2024-07-11T06:21:56.9020006Z = note: LINK : fatal error LNK1104: cannot open file 'C:\a\rust\rust\build\x86_64-pc-windows-msvc\stage1-rustc\x86_64-pc-windows-msvc\release\deps\rustc_main-aa69889b5edbcee3.exe'
2024-07-11T06:21:56.9021130Z
2024-07-11T06:21:56.9021508Z
2024-07-11T06:21:56.9021752Z
2024-07-11T06:21:56.9148094Z error: could not compile `rustc-main` (bin "rustc-main") due to 1 previous error
2024-07-12T10:55:07 49_auto - x86_64-msvc-ext.txt "../logs-rust-lang-ci-rust\\logs_9906744272.zip" found match https://github.com/rust-lang-ci/rust/actions/runs/9906744272
2024-07-12T11:55:13.9048987Z error: failed to remove file `C:\a\rust\rust\build\x86_64-pc-windows-msvc\stage2-tools\x86_64-pc-windows-msvc\release\cargo.exe`
2024-07-12T11:55:13.9049831Z
2024-07-12T11:55:13.9049952Z Caused by:
2024-07-12T11:55:13.9050247Z Access is denied. (os error 5)
2024-07-13T12:06:40 49_auto - x86_64-msvc-ext.txt "../logs-rust-lang-ci-rust\\logs_9919764593.zip" found match https://github.com/rust-lang-ci/rust/actions/runs/9919764593
2024-07-13T13:09:34.8085554Z error: failed to remove file `C:\a\rust\rust\build\x86_64-pc-windows-msvc\stage2-tools\x86_64-pc-windows-msvc\release\cargo.exe`
2024-07-13T13:09:34.8086897Z
2024-07-13T13:09:34.8087087Z Caused by:
2024-07-13T13:09:34.8087535Z Access is denied. (os error 5)
2024-07-15T02:21:06 52_auto - dist-x86_64-msvc.txt "../logs-rust-lang-ci-rust\\logs_9932760975.zip" found match https://github.com/rust-lang-ci/rust/actions/runs/9932760975
2024-07-15T03:17:43.2981649Z = note: LINK : fatal error LNK1104: cannot open file 'C:\a\rust\rust\build\x86_64-pc-windows-msvc\stage1-rustc\x86_64-pc-windows-msvc\release\deps\rustc_main-4e69e1e45706632e.exe'
2024-07-15T03:17:43.2982781Z
2024-07-15T03:17:43.2983034Z
2024-07-15T03:17:43.2983208Z
2024-07-15T03:17:43.3107182Z error: could not compile `rustc-main` (bin "rustc-main") due to 1 previous error
2024-07-16T05:02:37 49_auto - x86_64-msvc-ext.txt "../logs-rust-lang-ci-rust\\logs_9951025123.zip" found match https://github.com/rust-lang-ci/rust/actions/runs/9951025123
2024-07-16T06:08:12.4360760Z error: failed to remove file `C:\a\rust\rust\build\x86_64-pc-windows-msvc\stage2-tools\x86_64-pc-windows-msvc\release\cargo.exe`
2024-07-16T06:08:12.4362216Z
2024-07-16T06:08:12.4362481Z Caused by:
2024-07-16T06:08:12.4363081Z Access is denied. (os error 5)
2024-07-17T18:19:38 49_auto - x86_64-msvc-ext.txt "../logs-rust-lang-ci-rust\\logs_9979412052.zip" found match https://github.com/rust-lang-ci/rust/actions/runs/9979412052
2024-07-17T19:24:48.6223853Z error: failed to remove file `C:\a\rust\rust\build\x86_64-pc-windows-msvc\stage2-tools\x86_64-pc-windows-msvc\release\cargo.exe`
2024-07-17T19:24:48.6225041Z
2024-07-17T19:24:48.6225224Z Caused by:
2024-07-17T19:24:48.6225659Z Access is denied. (os error 5)
Starting 2024-07-02, https://github.com/rust-lang/rust/pull/127152 added a mitigation measure in bootstrap to cover the most common culprit of bootstrap attempting to delete an executable. However, there are several other programs that are still having problems, such as rustc itself, the msvc linker, and cargo.
I have tried using handle.exe, and the RestartManager API to try to detect if there is another process with an open handle on a file, but no success.
I have tried rolling back the source tree to 2024-06-25, and it still reproduces the problem (before we started seeing it in CI).
The last Windows image release before this started was https://github.com/actions/runner-images/releases/tag/win22%2F20240618.1. From what I can tell, this rolled out about 5 days earlier, during which there weren't any failures, but it is difficult to tell if that could be related.
The last stage0 bump was 2024-06-11, several weeks before it started.
https://github.com/actions/runner-images/issues/4086 is a similar issue we've had in the past, though we don't know what the fix was.
From what I can tell, this doesn't seem to be affecting the GNU builders.
Most recent https://github.com/rust-lang-ci/rust/actions/runs/9979412052/job/27578504068
You can also copy any of the https://github.com/rust-lang-ci/rust/actions/runs/... links in the logs above (sorry I didn't make them clickable). Just beware the logs get deleted (90 days?).
| PR | Failure cause |
|---|---|
| #128032 | LINK : fatal error LNK1104: cannot open file 'C:\a\rust\rust\build\x86_64-pc-windows-msvc\stage1-tools\x86_64-pc-windows-msvc\release\deps\miri-bdcbed7500be44cb.exe' |
| #128031 (1) | error: failed to remove file `C:\a\rust\rust\build\x86_64-pc-windows-msvc\stage2-tools\x86_64-pc-windows-msvc\release\cargo.exe |
| #128031 (2) | LINK : fatal error LNK1104: cannot open file 'C:\a\rust\rust\build\x86_64-pc-windows-msvc\stage1-rustc\x86_64-pc-windows-msvc\release\deps\rustc_main-4206913879fae405.exe' |
| #124895 | LINK : fatal error LNK1104: cannot open file 'C:\a\rust\rust\build\x86_64-pc-windows-msvc\stage1-rustc\x86_64-pc-windows-msvc\release\deps\rustc_main-cac7a0b14c781d14.exe' |
| #128270 | error: failed to remove file `C:\a\rust\rust\build\x86_64-pc-windows-msvc\stage2-tools\x86_64-pc-windows-msvc\release\cargo.exe`, Access is denied (os error 5) |
| #128577 | LINK : fatal error LNK1104: cannot open file 'C:\a\rust\rust\build\x86_64-pc-windows-msvc\stage1-rustc\x86_64-pc-windows-msvc\release\deps\rustc_main-4f38a2f26fa25d5a.exe' |
| #128037 (1) | LINK : fatal error LNK1104: cannot open file 'C:\a\rust\rust\build\x86_64-pc-windows-msvc\stage1-rustc\x86_64-pc-windows-msvc\release\deps\rustc_main-abe621a284360afc.exe' |
| #128037 (2) | LINK : fatal error LNK1104: cannot open file 'C:\a\rust\rust\build\x86_64-pc-windows-msvc\stage1-rustc\x86_64-pc-windows-msvc\release\deps\rustc_main-abe621a284360afc.exe' |
| #128936 | error: failed to remove file `C:\a\rust\rust\build\x86_64-pc-windows-msvc\stage2-tools\x86_64-pc-windows-msvc\release\cargo.exe`, Access is denied (os error 5) |
| #129054 | error: failed to remove file `C:\a\rust\rust\build\x86_64-pc-windows-msvc\stage2-tools\x86_64-pc-windows-msvc\release\cargo.exe`, Access is denied (os error 5) |
| #129066 | std\src\os\windows\process.rs - os::windows::process::CommandExt::raw_attribute (line 330) stdlib test failed which is very weird, called `Result::unwrap()` on an `Err` value: Os { code: 5, kind: PermissionDenied, message: "Access is denied." |
| #128742 | LINK : fatal error LNK1104: cannot open file 'C:\a\rust\rust\build\x86_64-pc-windows-msvc\stage1-rustc\x86_64-pc-windows-msvc\release\deps\rustc_main-abe621a284360afc.exe' |
| #128371 | LINK : fatal error LNK1104: cannot open file 'C:\a\rust\rust\build\x86_64-pc-windows-msvc\stage1-rustc\x86_64-pc-windows-msvc\release\deps\rustc_main-abe621a284360afc.exe' |
| #128677 | LINK : fatal error LNK1104: cannot open file 'C:\a\rust\rust\build\x86_64-pc-windows-msvc\stage1-rustc\x86_64-pc-windows-msvc\release\deps\rustc_main-abe621a284360afc.exe' |
| #128982 | error: failed to remove file `C:\a\rust\rust\build\x86_64-pc-windows-msvc\stage2-tools\x86_64-pc-windows-msvc\release\cargo.exe`, Access is denied (os error 5) |
| #128722 | error: failed to remove file `C:\a\rust\rust\build\x86_64-pc-windows-msvc\stage1-tools\x86_64-pc-windows-msvc\release\miri.exe`, Access is denied. (os error 5) |
| #129226 | error: failed to remove file `C:\a\rust\rust\build\x86_64-pc-windows-msvc\stage1-tools\x86_64-pc-windows-msvc\release\miri.exe`, Access is denied. (os error 5) |
| #127672 | error: failed to remove file `C:\a\rust\rust\build\x86_64-pc-windows-msvc\stage1-tools\x86_64-pc-windows-msvc\release\miri.exe`, Access is denied. (os error 5) |
| #129304 | LINK : fatal error LNK1104: cannot open file 'C:\a\rust\rust\build\x86_64-pc-windows-msvc\stage1-rustc\x86_64-pc-windows-msvc\release\deps\rustc_main-ff540feda8314309.exe' |
Another: https://github.com/rust-lang/rust/pull/128982#issuecomment-2295068323
It seems like a lot of these failures are specifically miri.exe for whatever reason, especially recently. Could we just adjust bootstrap to skip the rest of the step on MSVC once it gets to the point that it would remove this file? I'm not sure what exactly relies on this.
I'm planning to investigate some of the file removal logic in bootstrap. Working theory is some symlink removal related logic on Windows.
EDIT: some of the failures seem to be emitted from cargo, but bootstrap seems to also have its share
Crosslinking some ongoing Zulip discussion: https://rust-lang.zulipchat.com/#narrow/stream/246057-t-cargo/topic/Cargo.20investigation.3A.20High.20failure.20rate.20in.20rust-lang.20CI
I opened https://github.com/actions/runner-images/issues/10483 to see if GitHub is able to help us out.
I have been looking in to this. So to sum up, the major issue atm is that builds sometimes fail when overwriting a previous build. Specifically removing the old .exe files that are hard linked into the final artefact directory fails with access denied. The issue appears to affect windows-msvc builders but not windows-gnu. However, there are a number of differences there (not just gnu vs. msvc) so that doesn't necessarily narrow it down as much as you might hope (although it for sure helps).
One suggestion is that a process like vctip.exe or other msvc telemetry thing could be holding a lock on the file, preventing it from being deleted. I feel I've thoroughly ruled this possibility out by setting opt-out registry entries and even nuking the exes.
The issue is sporadic which does suggest it's the timing of... something... that leads to this problem. Maybe some process does linger in the background a little too long or maybe we've got an issue with multithreading builds. Maybe neither of those things. Anti-malware can interfere with builds while being outside the confines of our job objects so it's always a possible suspect.
I believe jieyouxu's test PR will help in tracking the issue back to its original source. One of my bootstrap workarounds ended up just punting the problem on so that it's hit by Cargo further down the line. We could add workarounds into Cargo too but it would be more useful to go in the opposite direction and look at where the problem starts.
One last thing, testing this has been slow going. I haven't yet replicated it locally and it takes a while in CI to run. So far my attempts at cutting down the code CI runs has reduced the failure rate, which doesn't help when I want it to fail.
Also it might be confirmation bias, but it sure feels like the failure rate is very high on rollups recently (maybe a week?), its at least once if not multiple times it feels like. On try jobs I can also easily obtain the failure by chance, which is good and bad.
There also seems to be multiple issues leading to such a high failure rate:
- The executable removal permission denied is the most common issue (or issues).
- Failing to create temp dir seems to crop up every so often as well, which I think is a different issue.
#129540 failure is interesting because it falls into the second category:
---- test::example_with_dev_dep stdout ----
running `C:\a\rust\rust\build\x86_64-pc-windows-msvc\stage2-tools\x86_64-pc-windows-msvc\release\cargo.exe test -v`
thread 'test::example_with_dev_dep' panicked at tests\testsuite\test.rs:2451:10:
test failed running `C:\a\rust\rust\build\x86_64-pc-windows-msvc\stage2-tools\x86_64-pc-windows-msvc\release\cargo.exe test -v`
error: process exited with code 101 (expected 0)
--- stdout
--- stderr
[trimmed]
error: couldn't create a temp dir: Access is denied. (os error 5) at path "C:\\a\\_temp\\msys64\\tmp\\rustcAND7yQ"
error: could not compile `foo` (example "ex") due to 1 previous error
Yeah it looks like (https://github.com/rust-lang/rust/pull/129522#issuecomment-2308644138) we can't delete e.g. C:\a\rust\rust\build\x86_64-pc-windows-msvc\stage1-tools-bin\miri.exe so we try to paper over it by renaming the executable, but I'm not sure if that's actually mitigating the problem that we can't remove the executable
2024-08-25T03:58:14.0023046Z [DEBUG] copy_link_internal: src=`C:\a\rust\rust\build\x86_64-pc-windows-msvc\stage1-tools\x86_64-pc-windows-msvc\release\miri.exe`, dst=`C:\a\rust\rust\build\x86_64-pc-windows-msvc\stage1-tools-bin\miri.exe`
2024-08-25T03:58:14.0025248Z [DEBUG] copy_link_internal: failed to remove file dst=`C:\a\rust\rust\build\x86_64-pc-windows-msvc\stage1-tools-bin\miri.exe`: Access is denied. (os error 5)
2024-08-25T03:58:14.0032786Z [DEBUG] copy_link_internal: can't delete, so renamed dst=`C:\a\rust\rust\build\x86_64-pc-windows-msvc\stage1-tools-bin\miri.exe` to tmp_filename=`C:\a\rust\rust\build\x86_64-pc-windows-msvc\stage1-tools-bin\miri.exe-1724558294002310100`
2024-08-25T03:58:14.0034684Z [DEBUG] copy_link_internal: saving symlink metadata from src=`C:\a\rust\rust\build\x86_64-pc-windows-msvc\stage1-tools\x86_64-pc-windows-msvc\release\miri.exe`
2024-08-25T03:58:14.0039400Z [DEBUG] copy_link_internal: created hard link src=`C:\a\rust\rust\build\x86_64-pc-windows-msvc\stage1-tools\x86_64-pc-windows-msvc\release\miri.exe` with dst=`C:\a\rust\rust\build\x86_64-pc-windows-msvc\stage1-tools-bin\miri.exe`
Also it might be confirmation bias, but it sure feels like the failure rate is very high on rollups recently (maybe a week?), its at least once if not multiple times it feels like. On try jobs I can also easily obtain the failure by chance, which is good and bad.
This definitely feels very true - we have to hitting the problem close to 50% of the time at this point, if not more. Doesn't seem to be limited to rollups though, the LLVM bump failed twice with this, my f16 PR failed three times, https://github.com/rust-lang/rust/pull/129295 is at two...
I wonder if it would be possible to run the runner-images scripts in a VM to reproduce the environment https://github.com/actions/runner-images/blob/d72a4523b15eb2e303c1ea5100a23c34d9475f4f/.github/workflows/windows2022.yml.
If this isn't improved by Monday (seems somewhat unlikely), we should see if infra can get in touch with GH to debug this better. (Not really worth hours of your effort trying to work around things they could get actual answers for)
What's really annoying is that I can't repro these issues locally either.
Specifically, it doesn't fail to remove miri.exe locally, so the rename fallback path is never reached anyway
this could be the same underlying issue as https://github.com/rust-lang/cargo/issues/11544 right? cross linking as I have not seen any references to the cargo ticket here yet
I was googling the error message and amusingly most top hits pointed back at the rust project in some way :smile:
this could be the same underlying issue as rust-lang/cargo#11544 right? cross linking as I have not seen any references to the cargo ticket here yet
It could be one of the causes causing one of the issues potentially yes. We're still trying to pin down the root cause for at least miri.exe removal permission denied issues.
Ok so the problem is that deleting a file (ultimately using DeleteFileW) sometimes returns ERROR_ACCESS_DENIED. This is a kind of catch-all error that encompasses a number of lower level status code, including (but not limited to):
STATUS_ACCESS_DENIEDSTATUS_DELETE_PENDINGSTATUS_FILE_RENAMEDSTATUS_CANNOT_DELETESTATUS_FILE_DELETED
In my tests, the underlying error is STATUS_CANNOT_DELETE. This may be caused by a race issue. Two operations occurring at the same moment may be running into issues. A workaround would be to simply retry the delete if it encounters ERROR_ACCESS_DENIED. Perhaps a better approach would be to figure out why it's happening in the first place and to stop doing that.
Cargo sets FILE_ATTRIBUTE_NOT_CONTENT_INDEXED on the target dir, but it looks like bootstrap doesn't for the build dir. I hope github doesn't enable indexing on the CI machines, but if they do then maybe that'd help.
I generated a process monitor .pml file. It can opened using procmon64.
However, I'm not yet seeing any obvious cause of the CANNOT DELETE failure.
Affects my PR #129019 so I thought I'd take a gander at the PML shared here.
There were two CANNOT_DELETE errors
One of them happened before any miri.exe started, the other one after the last miri.exe exited[1] so a process still running cannot be the root cause. Perhaps SysInternals Handles would show if there are any outstanding handles because of the file being open (the antivirus software might be a common guess)?
[1]:
"21:17:51,1536884","miri.exe","3568","Process Start"
"21:17:51,1886089","miri.exe","3568","Process Exit"
"21:17:51,3615827","miri.exe","6804","Process Start"
"21:17:51,3818521","miri.exe","6804","Process Exit"
"21:17:51,4726410","miri.exe","6372","Process Start"
"21:17:51,5007548","miri.exe","6372","Process Exit"
"21:17:51,5249692","miri.exe","6668","Process Start"
"21:17:51,5494028","miri.exe","6668","Process Exit"
"21:17:51,6786187","miri.exe","7952","Process Start"
"21:17:51,6973346","miri.exe","7952","Process Exit"
"21:17:51,9632364","miri.exe","5180","Process Start"
"21:18:13,9631696","miri.exe","5180","Process Exit"
"21:18:14,0437309","miri.exe","5128","Process Start"
"21:18:14,0876072","miri.exe","5128","Process Exit"
"21:18:14,1211076","miri.exe","6800","Process Start"
"21:18:14,1216762","miri.exe","6328","Process Start"
"21:18:15,0257685","miri.exe","6328","Process Exit"
"21:18:15,1537748","miri.exe","6800","Process Exit"
"21:18:15,1994376","miri.exe","4544","Process Start"
"21:18:15,2018318","miri.exe","8096","Process Start"
"21:18:15,2051426","miri.exe","1300","Process Start"
"21:18:15,2079730","miri.exe","5420","Process Start"
"21:18:15,2133517","miri.exe","2372","Process Start"
"21:18:15,2701454","miri.exe","4544","Process Exit"
"21:18:15,3118628","miri.exe","2796","Process Start"
"21:18:15,3503074","miri.exe","1300","Process Exit"
"21:18:15,4034604","miri.exe","2796","Process Exit"
"21:18:15,7402051","miri.exe","2372","Process Exit"
"21:18:15,7543653","miri.exe","5420","Process Exit"
"21:18:19,3894986","miri.exe","8096","Process Exit"
"21:18:19,4346246","miri.exe","2068","Process Start"
"21:18:19,4356985","miri.exe","1352","Process Start"
"21:18:19,4376855","miri.exe","6368","Process Start"
"21:18:19,5038631","miri.exe","2068","Process Exit"
"21:18:19,5141844","miri.exe","1352","Process Exit"
"21:18:19,5419264","miri.exe","6368","Process Exit"
"21:18:19,5427763","miri.exe","5356","Process Start"
"21:18:19,5444759","miri.exe","5556","Process Start"
"21:18:19,5489962","miri.exe","5016","Process Start"
"21:18:19,5517554","miri.exe","2888","Process Start"
"21:18:19,5549944","miri.exe","5320","Process Start"
"21:18:19,7685154","miri.exe","2888","Process Exit"
"21:18:20,0514524","miri.exe","5556","Process Exit"
"21:18:20,8660293","miri.exe","5016","Process Exit"
"21:18:24,0418653","miri.exe","5356","Process Exit"
"21:18:24,0789019","miri.exe","5964","Process Start"
"21:18:24,6157631","miri.exe","5964","Process Exit"
"21:18:26,7219583","miri.exe","5320","Process Exit"
"21:18:26,7639303","miri.exe","1392","Process Start"
"21:18:32,7115392","miri.exe","1392","Process Exit"
"21:18:32,7770281","miri.exe","7100","Process Start"
"21:18:32,7774907","miri.exe","5904","Process Start"
"21:18:32,8308202","miri.exe","7100","Process Exit"
"21:18:32,8607299","miri.exe","8092","Process Start"
"21:18:33,0002956","miri.exe","8092","Process Exit"
"21:18:33,0317539","miri.exe","5364","Process Start"
"21:18:33,3174302","miri.exe","5364","Process Exit"
"21:18:33,3484207","miri.exe","7484","Process Start"
"21:18:34,4177259","miri.exe","5904","Process Exit"
"21:18:34,4790770","miri.exe","7484","Process Exit"
"21:18:34,5161696","miri.exe","6680","Process Start"
"21:18:34,5636373","miri.exe","6680","Process Exit"
"21:18:34,5947541","miri.exe","8024","Process Start"
"21:18:34,6421069","miri.exe","8024","Process Exit"
"21:18:34,8585954","miri.exe","8056","Process Start"
"21:18:34,8779021","miri.exe","8056","Process Exit"
If the Handles tool doesn't reveal anything, a live kernel dump file a PS helper if no GUI installed or Task Manager (right click on the PID 4 and then "create kernel dump") would be the last resort. Goes w/o saying, the kernel dump should be shared if and only if the machine doesn't carry any sensitive information.
If the Handles tool doesn't reveal anything, a live kernel dump file a PS helper if no GUI installed or Task Manager (right click on the PID 4 and then "create kernel dump") would be the last resort. Goes w/o saying, the kernel dump should be shared if and only if the machine doesn't carry any sensitive information.
The CI machines probably have sensitive authorization tokens somewhere in their memory, unfortunately. Though T-infra is pretty diligent about the principle of least privilege, they still need to move some data around to some fairly specific buckets.
We have tried using Handles (and other methods) with no success. I think we've exhausted what we can do on our end. Any further investigation on the root cause will need to be handled by github. Either they fix it on their end or they tell us we're doing something wrong. In the meantime all we can do is attempt mitigations.
https://github.com/rust-lang/rust/pull/130569#issuecomment-2365034434 failure has a case where:
failed to copy
`C:\a\rust\rust\build\x86_64-pc-windows-msvc\stage0-tools\x86_64-pc-windows-msvc\release\miri.exe`
to
`C:\a\rust\rust\build\x86_64-pc-windows-msvc\stage0-tools-bin\miri.exe`:
The process cannot access the file because it is being used by another process. (os error 32)
in addition to a different failure
error: failed to remove file `C:\a\rust\rust\build\x86_64-pc-windows-msvc\stage1-tools\x86_64-pc-windows-msvc\release\miri.exe`
so there's definitely something stepping over something
Small update; some Cargo.lock updates were hitting a 100% success rate and failing to merge due to these sort of failures on MSVC jobs. The problem was identified to be a change to cc that was released in 1.0.101 only a couple days before we started to notice this issue. None of our lockfiles contained the relevant version until later, but the timing seems too convenient to be coincidence (maybe we were using an unlocked version somewhere).
Anyway, Chris fixed that issue at https://github.com/rust-lang/cc-rs/pull/1215 and that was released in 1.1.22. We just finished bumping cc to that version in all the lockfiles (except for RA and a Clippy test). It still isn't certain whether the cc bug caused the problem in the first place or if it just made some other problem worse, time to experiment.
Edit: nope, not solved :( https://github.com/rust-lang/rust/pull/131133#issuecomment-2387656490
~~I think https://github.com/rust-lang/rust/pull/133097#issuecomment-2480542107 is another kind of msvc spurious failure, my guess is some kind of malware scanner interfered with cdb...~~ nope genuine failure unrelated.