.cargo/bin in PATH points to different users' home directory

[thbusch]

Description

When using (Rust-) cargo install <binary> in a build job and then trying to run it, the executable cannot be found because the entry for the Cargo bin directory in PATH points to a different user (Default) than the user the build step is run as (VssAdministrator).

This issue happens sporadically, some builds with the same runner version don't have that issue (users in PATH and current user are the same)

Platforms affected

• [X] Azure DevOps
• [ ] GitHub Actions - Standard Runners
• [ ] GitHub Actions - Larger Runners

Runner images affected

• [ ] Ubuntu 18.04
• [ ] Ubuntu 20.04
• [ ] Ubuntu 22.04
• [ ] macOS 10.15
• [ ] macOS 11
• [ ] macOS 12
• [ ] Windows Server 2019
• [X] Windows Server 2022

Image version and build link

20221002.2

Is it regression?

na

Expected behavior

Entry for .cargo/bin in PATH should point to the same user home directory as the one that the build scripts are run as.

Actual behavior

User name in directory in PATH and current user name differ.

Repro steps

1. add a build script which adds a dependency via cargo install, for example cargo install cargo2junit
2. try to use the previously added dependency -> dependency cannot be found

[al-cheb]

Hey @thbusch. Could you attach logs with Default user?

[cry-inc]

I observed the same issues two days ago and reported it here: https://developercommunity.visualstudio.com/t/Hosted-Agent-Rust-Issue-on-Windows-2022/10164368 and was now told to come here.

For us, it happens only on one agent machine with the name "WIN-EERHFLKC3PP", all others with the same agent image work without issues. Our organization is in the EU-West region.

I observed the same things as Thomas. On the machine where it fails, PATH is C:\agents\2.211.0\externals\git\cmd;C:\agents\2.211.0\externals\git\mingw64\bin;C:\Program Files\MongoDB\Server\5.0\bin;C:\aliyun-cli;C:\vcpkg;C:\Program Files (x86)\NSIS\;C:\tools\zstd;C:\Program Files\Mercurial\;C:\hostedtoolcache\windows\stack\2.9.1\x64;C:\cabal\bin;C:\\ghcup\bin;C:\tools\ghc-9.4.2\bin;C:\Program Files\dotnet;C:\mysql\bin;C:\Program Files\R\R-4.2.1\bin\x64;C:\SeleniumWebDrivers\GeckoDriver;C:\Program Files (x86)\sbt\bin;C:\Program Files (x86)\GitHub CLI;C:\Program Files\Git\bin;C:\Program Files (x86)\pipx_bin;C:\npm\prefix;C:\hostedtoolcache\windows\go\1.17.13\x64\bin;C:\hostedtoolcache\windows\Python\3.9.13\x64\Scripts;C:\hostedtoolcache\windows\Python\3.9.13\x64;C:\hostedtoolcache\windows\Ruby\3.0.4\x64\bin;C:\tools\kotlinc\bin;C:\hostedtoolcache\windows\Java_Temurin-Hotspot_jdk\8.0.345-1\x64\bin;C:\Program Files\ImageMagick-7.1.0-Q16-HDRI;C:\Program Files (x86)\Microsoft SDKs\Azure\CLI2\wbin;C:\ProgramData\kind;C:\Program Files\Microsoft\jdk-11.0.12.7-hotspot\bin;C:\Windows\system32;C:\Windows;C:\Windows\System32\Wbem;C:\Windows\System32\WindowsPowerShell\v1.0\;C:\Windows\System32\OpenSSH\;C:\Program Files\dotnet\;C:\ProgramData\Chocolatey\bin;C:\Program Files\PowerShell\7\;C:\Program Files\Microsoft\Web Platform Installer\;C:\Program Files\Microsoft SQL Server\Client SDK\ODBC\170\Tools\Binn\;C:\Program Files\Microsoft SQL Server\150\Tools\Binn\;C:\Program Files\OpenSSL\bin;C:\Strawberry\c\bin;C:\Strawberry\perl\site\bin;C:\Strawberry\perl\bin;C:\ProgramData\chocolatey\lib\pulumi\tools\Pulumi\bin;C:\Program Files\TortoiseSVN\bin;C:\Program Files\CMake\bin;C:\ProgramData\chocolatey\lib\maven\apache-maven-3.8.6\bin;C:\Program Files\Microsoft Service Fabric\bin\Fabric\Fabric.Code;C:\Program Files\Microsoft SDKs\Service Fabric\Tools\ServiceFabricLocalClusterManager;C:\Program Files\nodejs\;C:\Program Files\Git\cmd;C:\Program Files\Git\mingw64\bin;C:\Program Files\Git\usr\bin;C:\Program Files\GitHub CLI\;c:\tools\php;C:\Program Files (x86)\sbt\bin;C:\SeleniumWebDrivers\ChromeDriver\;C:\SeleniumWebDrivers\EdgeDriver\;C:\Program Files\Amazon\AWSCLIV2\;C:\Program Files\Amazon\SessionManagerPlugin\bin\;C:\Program Files\Amazon\AWSSAMCLI\bin\;C:\Program Files\Microsoft SQL Server\130\Tools\Binn\;C:\Program Files\LLVM\bin;C:\Users\Default\.cargo\bin;C:\Users\VssAdministrator\AppData\Local\Microsoft\WindowsApps;C:\Users\packer\AppData\Roaming\npm;C:\Users\packer\AppData\Roaming\Composer\vendor\bin;C:\Users\VssAdministrator\.dotnet\tools;C:\ProgramData\chocolatey\lib\mingw\tools\install\mingw64\bin;C:\Users\packer\AppData\Roaming\cabal\bin;C:\tools\ghc-9.0.2\bin;C:\tools\ghc-9.2.4\bin;C:\ghcup\bin

On the other machines where it works its C:\agents\2.211.0\externals\git\cmd;C:\agents\2.211.0\externals\git\mingw64\bin;C:\Program Files\MongoDB\Server\5.0\bin;C:\aliyun-cli;C:\vcpkg;C:\Program Files (x86)\NSIS\;C:\tools\zstd;C:\Program Files\Mercurial\;C:\hostedtoolcache\windows\stack\2.9.1\x64;C:\cabal\bin;C:\\ghcup\bin;C:\tools\ghc-9.4.2\bin;C:\Program Files\dotnet;C:\mysql\bin;C:\Program Files\R\R-4.2.1\bin\x64;C:\SeleniumWebDrivers\GeckoDriver;C:\Program Files (x86)\sbt\bin;C:\Program Files (x86)\GitHub CLI;C:\Program Files\Git\bin;C:\Program Files (x86)\pipx_bin;C:\npm\prefix;C:\hostedtoolcache\windows\go\1.17.13\x64\bin;C:\hostedtoolcache\windows\Python\3.9.13\x64\Scripts;C:\hostedtoolcache\windows\Python\3.9.13\x64;C:\hostedtoolcache\windows\Ruby\3.0.4\x64\bin;C:\tools\kotlinc\bin;C:\hostedtoolcache\windows\Java_Temurin-Hotspot_jdk\8.0.345-1\x64\bin;C:\Program Files\ImageMagick-7.1.0-Q16-HDRI;C:\Program Files (x86)\Microsoft SDKs\Azure\CLI2\wbin;C:\ProgramData\kind;C:\Program Files\Microsoft\jdk-11.0.12.7-hotspot\bin;C:\Windows\system32;C:\Windows;C:\Windows\System32\Wbem;C:\Windows\System32\WindowsPowerShell\v1.0\;C:\Windows\System32\OpenSSH\;C:\Program Files\dotnet\;C:\ProgramData\Chocolatey\bin;C:\Program Files\PowerShell\7\;C:\Program Files\Microsoft\Web Platform Installer\;C:\Program Files\Microsoft SQL Server\Client SDK\ODBC\170\Tools\Binn\;C:\Program Files\Microsoft SQL Server\150\Tools\Binn\;C:\Program Files\OpenSSL\bin;C:\Strawberry\c\bin;C:\Strawberry\perl\site\bin;C:\Strawberry\perl\bin;C:\ProgramData\chocolatey\lib\pulumi\tools\Pulumi\bin;C:\Program Files\TortoiseSVN\bin;C:\Program Files\CMake\bin;C:\ProgramData\chocolatey\lib\maven\apache-maven-3.8.6\bin;C:\Program Files\Microsoft Service Fabric\bin\Fabric\Fabric.Code;C:\Program Files\Microsoft SDKs\Service Fabric\Tools\ServiceFabricLocalClusterManager;C:\Program Files\nodejs\;C:\Program Files\Git\cmd;C:\Program Files\Git\mingw64\bin;C:\Program Files\Git\usr\bin;C:\Program Files\GitHub CLI\;c:\tools\php;C:\Program Files (x86)\sbt\bin;C:\SeleniumWebDrivers\ChromeDriver\;C:\SeleniumWebDrivers\EdgeDriver\;C:\Program Files\Amazon\AWSCLIV2\;C:\Program Files\Amazon\SessionManagerPlugin\bin\;C:\Program Files\Amazon\AWSSAMCLI\bin\;C:\Program Files\Microsoft SQL Server\130\Tools\Binn\;C:\Program Files\LLVM\bin;C:\Users\VssAdministrator\.dotnet\tools;C:\Users\VssAdministrator\.cargo\bin;C:\Users\VssAdministrator\AppData\Local\Microsoft\WindowsApps

Looks like cargo/rustup/rust binaries are installed in a different folder?

Our hacky workaround is currently to detected the problematic agent by name and download and reinstall rustup manually before running any rust/rustup/cargo commands.

[cry-inc]

Here is an example job where the problem is visible:

  - job:
    pool:
      vmImage: windows-2022
    steps:
    - script: rustup update
      continueOnError: true
    - powershell: which rustup
    - powershell: which rustc
    - powershell: echo $env:PATH

For the faulty agent the first script will fail with exit code 1. I have attached the logs of two runs, one from our affected machine and one that worked because it was executed on another machine (with the same image).

not-working.txt working.txt

[al-cheb]

@cry-inc, could you please attach output - reg query HKCU\Environment and dir %USERPROFILE% from a failed build ?

[cry-inc]

reg query HKCU\Environment:

HKEY_CURRENT_USER\Environment
    Path    REG_EXPAND_SZ    C:\Users\Default\.cargo\bin;%USERPROFILE%\AppData\Local\Microsoft\WindowsApps;C:\Users\packer\AppData\Roaming\npm;C:\Users\packer\AppData\Roaming\Composer\vendor\bin;%USERPROFILE%\.dotnet\tools;C:\ProgramData\chocolatey\lib\mingw\tools\install\mingw64\bin;C:\Users\packer\AppData\Roaming\cabal\bin;C:\tools\ghc-9.0.2\bin;C:\tools\ghc-9.2.4\bin;C:\tools\ghc-9.4.2\bin;;C:\ghcup\bin;C:\Program Files\Mercurial\;
    TEMP    REG_EXPAND_SZ    %USERPROFILE%\AppData\Local\Temp
    TMP    REG_EXPAND_SZ    %USERPROFILE%\AppData\Local\Temp
    ChocolateyLastPathUpdate    REG_SZ    133092054979450164
    ChocolateyToolsLocation    REG_SZ    C:\tools
    _MSYS2_BASH    REG_SZ    C:\msys64\usr\bin\bash.exe
    _MSYS2_PREFIX    REG_SZ    x86_64
    GHCUP_INSTALL_BASE_PREFIX    REG_SZ    C:\
    CABAL_DIR    REG_SZ    C:\cabal
    VSTS_AGENT_PERFLOG    REG_SZ    C:\agents\perflog

dir %USERPROFILE%:

 Volume in drive C is Windows
 Volume Serial Number is 0E02-5BDE

 Directory of C:\Users\VssAdministrator

10/07/2022  12:35 PM    <DIR>          .
10/04/2022  08:38 AM    <DIR>          ..
10/02/2022  05:40 PM    <DIR>          .aliyun
10/02/2022  04:27 PM    <DIR>          .android
10/02/2022  04:00 PM    <DIR>          .azcopy
10/02/2022  04:45 PM    <DIR>          .azure
10/02/2022  03:52 PM    <DIR>          .azure-devops
10/02/2022  04:22 PM    <DIR>          .config
10/04/2022  08:42 AM    <DIR>          .docker
10/02/2022  05:17 PM    <DIR>          .dotnet
10/02/2022  04:03 PM    <DIR>          .gradle
10/02/2022  04:02 PM    <DIR>          .m2
10/02/2022  06:12 PM    <DIR>          .minikube
10/02/2022  04:23 PM                37 .npmrc
10/02/2022  04:00 PM    <DIR>          .pulumi
10/07/2022  12:35 PM    <DIR>          .rustup
10/02/2022  05:00 PM    <DIR>          .sbt
10/02/2022  03:37 PM    <DIR>          .templateengine
10/04/2022  08:43 AM    <DIR>          3D Objects
10/04/2022  08:43 AM    <DIR>          Contacts
10/04/2022  08:43 AM    <DIR>          Desktop
10/04/2022  08:43 AM    <DIR>          Documents
10/02/2022  05:14 PM            66,853 dotnet-install.ps1
10/04/2022  08:43 AM    <DIR>          Downloads
10/04/2022  08:43 AM    <DIR>          Favorites
10/04/2022  08:43 AM    <DIR>          Links
10/04/2022  08:43 AM    <DIR>          Music
10/04/2022  08:43 AM    <DIR>          Pictures
10/02/2022  05:00 PM    <DIR>          project
10/04/2022  08:43 AM    <DIR>          Saved Games
10/04/2022  08:43 AM    <DIR>          Searches
10/04/2022  08:41 AM    <DIR>          source
10/02/2022  05:00 PM    <DIR>          target
10/04/2022  08:43 AM    <DIR>          Videos
10/02/2022  06:15 PM    <DIR>          _bazel_packer
               2 File(s)         66,890 bytes
              33 Dir(s)  100,581,679,104 bytes free

job.txt

[al-cheb]

@cry-inc, could you please attach planId for the job above?

Click View raw log button, remove at the end of the link /logs/XXX(_apis/build/builds/2470053/logs/13 -> _apis/build/builds/2470053 ):

[cry-inc]

The planId for that job is 1f676ead-a3d4-4d31-a5bf-fd49e19a69ea

[al-cheb]

@thbusch, @cry-inc, Could you please check if the issue still exists on 2.0.91.1 image provisioner?

[al-cheb]

Feel free to reopen the issue if you have any concerns.

[cry-inc]

@al-cheb Our builds get no longer the machine WIN-EERHFLKC3PP assigned, so its impossible to say if it was fixed. I cannot reproduce the issue with any other agents that used the 2.0.91.1 image provisioner. Thanks!