Program 'winget.exe' failed to run: The file cannot be accessed by the system.At line:1 char:1 [Running Virtual Machine via SSH]

[ranavale]

Brief description of your issue

I created a VM and I installed winget using latest release on its github page, then I connected to my VM from VSCode SSH remote connection. I am trying to run the winget from my local remote-VSCode (connected to VM via SSH). Program 'winget.exe' failed to run: The file cannot be accessed by the system. At line:1 char:1 [Running Virtual Machine via SSH]

Steps to reproduce

run comment "winget"

Expected behavior

winget to work This is output when I login to my VM and run winget

Actual behavior

Program 'winget.exe' failed to run: The file cannot be accessed by the system. At line:1 char:1 [Running Virtual Machine via SSH]

Environment

Winget Package

Windows Package Manager v1.0.11692
Copyright (c) Microsoft Corporation. All rights reserved.

Windows specifications:

Edition: Windows 10 Enterprise for Virtual Desktops
Version: 21H1
OS Build: 19043.1237

App Installer

Version: 1.12.11692.0

[jedieaston]

This is a similar issue to https://github.com/microsoft/winget-cli/issues/995.

If it helps the team narrow it down, it doesn't just happen with winget.exe, but with any App Execution Alias (Windows Terminal registers wt.exe and it doesn't work either). It happens over PowerShell Remoting and SSH (via the Windows built-in OpenSSH Server), and it happens even if winget works completely fine on the local system from a shell. It also happens if you are running Windows inside a docker container and attempting to execute winget via docker exec.

This is probably not a winget problem then, but @denelon can you ask whoever manages the App Execution Aliases stuff to look into it? It basically blocks using winget remotely unless you get creative.

Proofsies:

PS E:\> Enter-LabPSSession wingettest

[wingettest]: PS C:\Users\Administrator\Documents> whoami 
wingettest\administrator

[wingettest]: PS C:\Users\Administrator\Documents> winget --info
Program 'winget.exe' failed to run: The file cannot be accessed by the system.
    + CategoryInfo          : ResourceUnavailable: (:) [], ApplicationFailedException
    + FullyQualifiedErrorId : NativeCommandFailed
 

[wingettest]: PS C:\Users\Administrator\Documents> python3
Program 'python3.exe' failed to run: The file cannot be accessed by the system.
    + CategoryInfo          : ResourceUnavailable: (:) [], ApplicationFailedException
    + FullyQualifiedErrorId : NativeCommandFailed
 

[wingettest]: PS C:\Users\Administrator\Documents> python
Program 'python.exe' failed to run: The file cannot be accessed by the system.
    + CategoryInfo          : ResourceUnavailable: (:) [], ApplicationFailedException
    + FullyQualifiedErrorId : NativeCommandFailed
 

[wingettest]: PS C:\Users\Administrator\Documents> wt 
Program 'wt.exe' failed to run: The file cannot be accessed by the system.
    + CategoryInfo          : ResourceUnavailable: (:) [], ApplicationFailedException
    + FullyQualifiedErrorId : NativeCommandFailed
 

[wingettest]: PS C:\Users\Administrator\Documents> # Aw, beans!

[JohnMcPMS]

@jedieaston , my guess is that this is because PowerShell is attempting to read the file before launching it. The way that AEA is implemented in the OS makes the shells not work if they attempt to read it, since it isn't the actual file (nor any kind of file really).

There are a few solutions:

1. We change a few things about the winget package so that the actual binary is launchable directly. As I recall, there was some work done in more recent Windows builds to enable that. That would possibly enable a workaround that an Invoke-WinGet PowerShell function could be written to handle. All speculative on my part; only helps those who seek out the solution but I imagine this type of remoting falls into "people who seek out solutions" territory anyway. Only works on "newer" (TBD) OS.
2. PowerShell (and docker, and anyone else) updates to not attempt to read from the "executable" or at least handle a failure to do so by simply executing it. Requires multiple teams to do work; will keep coming up.
3. The OS is updated to allow the file to be read through the AEA. Only works in some future timeframe, but would hopefully solve the issue for everything. I think might actually be possible (I was party to the original implementation but am no longer on the owning team). Requires getting that work approved, and it is not a bug level change.

Our changes for 1 are actually pretty easy, but again it would only (potentially) make something like this work:

C:\Program Files\WindowsApps\Microsoft.DesktopAppInstaller_1.16.12581.0_x64__8wekyb3d8bbwe> winget.exe install foo

And not the default path winget command.

[jedieaston]

Obviously 3 would be wonderful, but if we have to settle for 1 that’d still be good enough to get winget working for now in these scenarios.

By TBD, do you mean that the OS changes necessary for 1 aren’t in stable yet? The only thing I know of that rings a bell with that is Invoke-CommandInDesktopPackage and that doesn’t seem to work either.

[JohnMcPMS]

Agreed on 3, I think there is a case for it but if it is going to take tens of hours of work then there needs to be a justification that meets the bar. Need to ensure that it is actually the problem first, then we can put it on the backlog. But even if it was a go today, I don't think we would see it released for at least a year and then it wouldn't be serviced back to older Windows versions. We need the short term solution too.

By TBD I meant more "to be looked up because I remember seeing the PR but I have no idea when it was released, or if it even has been yet". This was work specifically to enable creating the process directly on the package file in certain instances. Normally packaged processes need to be launched by special API to enable all of packagy goodness (don't @ me), but this provided the ability to CreateProcess against the file and have it invoke the special API for you.

[ghost]

ah, I see everyone is busy with App Execution Aliases , but that's not the issue. the issue is MSIX packaged apps. MSIX packages can't be run over SSH or WinRM , even worse one user can't launch the app as another user. MSIX doesn't support per machine class registration.

[jedieaston]

But if I SSH as myself into the system, the MSIX apps I've installed as myself work. The AEAs don't though.

[ghost]

TL;DR : win32 OpenSSH port might want to implement AEAs/NTFS reparse points handling mechanism. I believe, @JohnMcPMS you as a MS employee, your words/winget teams words will have more weight to the win32-openssh team than ours. people even have pointed out where the fix might be needed : https://github.com/PowerShell/openssh-portable/blob/0b73c4636d38f0c69424721d52d0e7752db99c81/contrib/win32/win32compat/fileio.c#L1124

Does it ? 🤔 No MSIX (or Appx) packages regardless of installation medium (loose appxmanifest.xml installation or through AppInstaller installation or directly through Store Installation) run over SSH, here on my machine.

here's a screenshot of me from a non-elevated shell sshing to myself and trying to launch a MSIX package over ssh (eg: winget)

crap, let me dig a little

Update: same result regardless of specified locations or AEAs. It turns out AEA is the MSIX Problem, which could be anything among Symbolik Link/Junction/Mount Point/Reparse Point. is indeed a Special NTFS Reparse Point. specified path(%program files%\windowsapps\CrapFolder\crap.exe launches fine over SSH)

[Masamune3210]

Here you are trying to use a AEA, proving the point. Winget should work if you drill down into the correct package location and find the specific exe

[ghost]

Here you are trying to use a AEA

yeah, that's right ~\appdata\local\microsoft\windowsapps is the AEA folder.

[Masamune3210]

Oooh i see, I didnt realize that AEA's were plopped in a specific folder, figured they were just namespace links or some other magic. Forgive the comment

[jedieaston]

@denelon Do you know how winget is executed during the Installation Validation step of the winget-pkgs pipeline? Presumably it isn't via PowerShell Remoting or SSH due to this bug.

[JohnMcPMS]

@mshaikhcool , do you have an issue that I could comment on? I don't really understand why these remoting shell type operations need to read the target "exe", but I can certainly comment on what could be done in this case. I'm not sure that the internal data of the AEA reparse point are public, but the type certainly is (IO_REPARSE_TAG_APPEXECLINK).

@jedieaston , @ChuckFerring would know for sure but I think it is part of a script run on the machine as remoting would not be allowed for security reasons.

[ChuckFerring-zz]

@jedieaston, you are correct. During validation the script is run on the validation machine. It is not done via PowerShell remoting or SSH.

[matsmcp]

May I suggest 4. Repackage with a classic MSI/EXE instead of using "modern" packaging? It would also fix the Issue with non internet connected installations at the same time

[roland-5]

I can confirm, with new preview version, I was able to use ssh on virtual machine with windows, update and install applications.

[denelon]

Can anyone else confirm? If this is fixed, I would like to close this issue.

[ranavale]

I am still seeing the issue, (Note: I am connecting to my VM using VSCode SSH feature)

[roland-5]

I use Arch Linux. As my terminal Alacritty, use KVM for virtualization, and installed Windows 11 and use this for ssh server in Windows 11 VM.

Hmm. I created new VM with Windows 11 and using in this VM Windows terminal I was able to ssh log in to first Windows 11 VM and without problem installed some programs (vim, libreoffice).

EDIT. OK. I didn't saw you exactly use VSCode. I previously tried installing remotely apps, and have the same warning like you. As I see, VSCode have this problem still. Sorry for misunderstanding.

[denelon]

@ranavale does this fail only with SSH in VSCode? I'm not sure what the differences are between that and the ssh server @roland-rollo is using.

[jedieaston]

It fails with any ssh/remoting/docker exec session due to the issue that @JohnMcPMS described around AEAs not working. Extracting AppInstallerCLI.exe and the requisite dependencies (resources, dlls, etc) are the only way I've gotten it to work so far.

[A9G-Data-Droid]

I'm confused by this whole AEA problem when it seems like filesystem hardlinks would be the solution. Why invent some new type of file link that doesn't work when filesystem hardlinks have worked as advertised since the early days of Unix?

[ghost]

You will be able understand once you start to see the world without your "Unix and Unix Clones" Glass.

see https://github.com/microsoft/winget-cli/discussions/1838#discussioncomment-1950473

mainly, it's an issue because of the lack of interest of SSH/WinRM/Docker client programs.

[DylanWoodrow]

I am also still having this issue with build 1.17.3411.0 on 21H2. Notably when running Powershell as admin.

[roland-5]

Hmm, I tried this at computers with Windows 10 at work, and they have this same error message as in ranavale and ghost screenshots. EDIT. I tried few Windows 10 builds, and on everyone is this problem. On Windows 11 is not problem.

[dawin-knipp]

@DylanWoodrow can confirm, happens to me even on my local machine, no ssh / remoting involved. When running as admin it fails, when running as currently logged in user, it works (but can't make real use of winget as regular user - UAC on every updated package). Also doesn't matter if I use powershell or cmd, same result (with different error message: The system cannot execute the specified program).

[ghost]

@dawin-knipp That's one of the major limitations of the MSIX packaging format, has nothing to do with winget-itself. see https://github.com/microsoft/winget-cli/pull/1841#issuecomment-1009193824

[A9G-Data-Droid]

@dawin-knipp That's one of the major limitations of the MSIX packaging format, has nothing to do with winget-itself. see #1841 (comment)

So can we get an MSI installer? It's a hard requirement until MSIX packages are fixed.

[ghost]

IDK. That's been asked like a thousand times before. To be brutally honest, they are reluctant to ditch MSIX.

[jedieaston]

It's not really MSIX that is the problem, it's the use of a App Execution Alias. You can create MSIX packages that plop "real" executables in the right places, but that comes with other tradeoffs (for one, you'll have a bunch of copies of winget installed under every profile instead of one in the system directory).

[A9G-Data-Droid]

Naturally, the installer would need to be configured properly. There needs to be two options:

1. Per user install
2. Admin install

If you're an Admin trying to update all software in the system, you obviously need admin rights and a system install. The current user install may have it's uses but I don't know what they are. Doesn't work for my use case, which is to use it like a Windows package manager. (Wasn't that the point?)