[Bug] Shim breaks interoperability between restic and rclone

[Zottelchen]

Bug Report

Current Behavior

When attempting to use restic backups with rclone (both installed via scoop) it does not work with error messages like: Fatal: unable to open repo at rclone:onedrive:snapshot: error talking HTTP to rclone: Get "http://localhost/file-5577006791947779410": unexpected EOF

Expected Behavior

restic should be able to work, like it does with a non-scoop install.

Additional context/output

Scoop is "shimming" apps installed via https://github.com/ScoopInstaller/Shim/blob/master/src/shim.cs . However, for unknown reason, the stdin/stdout handles which should be inherited from shim process (set by restic by redirecting) aren't passed to rclone process properly if shim process is started with DETACHED_PROCESS. Even when I modified shim.cs to manually pass handles, the issue persists. I observed that a blank console window flashed which may indicate a new console was allocated.

Originally posted by @cqjjjzr in https://github.com/restic/restic/issues/3681#issuecomment-1081441509

Possible Solution

—

System details

Windows version: 10

OS architecture: 64bit

PowerShell version: 5.1.19041.1320

Additional software: restic, rclone

Scoop Configuration

{
    "lastupdate":  "2022-03-28T14:42:06.1230313+02:00",
    "SCOOP_REPO":  "https://github.com/lukesampson/scoop",
    "SCOOP_BRANCH":  "master"
}

[cqjjjzr]

All 3 shim implementations (kiennq, 71 and C#) failed to work.

Additional information: On Windows, restic is currently creating process with DETACHED_PROCESS creation flag. Communication between rclone and restic relies on pipes, and they are established via mechanisms provided by golang. Since I'm not familiar with golang, I just did some simple digging. rclone uses STD_INPUT_HANDLE as its stdin stream. However, at the restic side, golang is using some weird way to pipe data to/from stdio streams (see https://cs.opensource.google/go/go/+/refs/tags/go1.18:src/os/exec/exec.go;drc=refs%2Ftags%2Fgo1.18;l=244 ) and I can't figure it out completely now.

upd: As seen in https://cs.opensource.google/go/go/+/master:src/syscall/exec_windows.go;l=368 , the stdio handles are passed to child process via hStdInput, hStdOutput and hStdError fields of STARTUPINFOEX / STARTUPINFOW structure.

Therefore, it seems that shim failed to pass some sort of stdio handles to the child (rclone in this case) process.

[rashil2000]

Scoop currently uses the kiennq shim by default. Can you try opening an issue about this at https://github.com/kiennq/scoop-better-shimexe?

[Zottelchen]

Literally can't since issues are disabled for that repository. Hopefully, @kiennq receives notifications ;)

[cqjjjzr]

I managed to pass the handles properly by modifying the shim implementation. Will open a PR in the kiennq repo later. But I noticed the kiennq repo hasn't been updated for over 1 year...

~~and the mod didn't work in restic... maybe my test is improperly written~~

~~to my great surprise, it works on my system but not in my testing virtual machine.~~

It works :) although a blank console window still appears..

[rashil2000]

I noticed the kiennq repo hasn't been updated for over 1 year...

@issaclin32 should we fork the repo?

[issaclin32]

@issaclin32 should we fork the repo?

I think we should.

[rashil2000]

/cc @niheaven @rasa

[kiennq]

Literally can't since issues are disabled for that repository. Hopefully, @kiennq receives notifications ;)

Sorry, I didn't notice the issues are being disabled on that repo. Re-enable it now

[niheaven]

Transfer repos to ScoopInstaller and don't change the ownership?

[kiennq]

Transfer repos to ScoopInstaller and don't change the ownership?

That's brilliant. I don't have permission to create public repositories on ScoopInstaller now, so how to I transfer it?

[niheaven]

Okay，I'll invite you as maintainer, and then you could transfer it. Please check the invitation later.

[kiennq]

@niheaven I still don't have permission to create a public repository on ScoopInstaller (I have merge permission, but don't have permission to create a new repository). But worry not, I've requested to transfer the kiennq/scoop-better-shimexe repo to you, so please put it to the ScoopInstaller repo later :)

[chawyehsu]

I suggest putting all shim implementations into https://github.com/ScoopInstaller/Shim so we can use CI/CD to deliver different shim binaries concurrently. If you @niheaven @kiennq agree with it, I could help to refactor code structure.

[kiennq]

I suggest putting all shim implementations into https://github.com/ScoopInstaller/Shim so we can use CI/CD to deliver different shim binaries concurrently. If you @niheaven @kiennq agree with it, I could help to refactor code structure.

Sure, please go ahead with that. Note that I've also added the CI/CD to my repro to automatically build and publish a new version, hope it can help with the current CI/CD

[niheaven]

Go ahead @chawyehsu 🎉

I'm thinking of some type of plugin system that several types of shim or other additional features should be added into Scoop Core after installation in need. The plugins are different choices or some performance improvements. I don't know if it is a good idea to be added to Scoop since this may increase some code/UX complexity.

[viceice]

I seeing the same issue with terraform. no interactive input possible. ctrl+c is also not working

btw: I'm using cmder as console host with powershell v7 on windows server 2016

[rashil2000]

Filing this under #1606