ScoopInstaller
[Bug] Installing specific version fails due to unrelated architecture not supported for that version
Bug Report
Current Behavior
Example: the current nodejs manifest (20.4.0) contains a download link for the arm64 architecture, which nodejs 14.21.3 did/does not provide. When running e.g. scoop install [email protected] it tries to autogenerate the manifest, but fails (even on a non-arm architecture!) because it cannot get a hash for the missing arm64 binary.
Expected Behavior
Scoop should autogenerate the manifest only for the system's architecture and skip (or at least not fail on) all other architectures.
Additional context/output
This issue is not NodeJS specific, but relates to an edge case in the autogeneration of the manifest. The particular example (NodeJS 14) can of course also successfully be installed using the versions bucket.
$ scoop install [email protected]
WARN Given version (14.21.3) does not match manifest (20.4.0)
WARN Attempting to generate manifest for 'nodejs' (14.21.3)
Autoupdating nodejs
Searching hash for node-v14.21.3-win-x86.7z in https://nodejs.org/dist/v14.21.3/SHASUMS256.txt.asc
Found: 7a1f86386bb5f39c93bfd910a10966c69331c3f775457c0ea0c044ee6fbf000f using Extract Mode
Searching hash for node-v14.21.3-win-x64.7z in https://nodejs.org/dist/v14.21.3/SHASUMS256.txt.asc
Found: a00c71e662cb1752f180e54921ca2c79bdefc7200a51b8805888ac3723889579 using Extract Mode
Searching hash for node-v14.21.3-win-arm64.7z in https://nodejs.org/dist/v14.21.3/SHASUMS256.txt.asc
Could not find hash in https://nodejs.org/dist/v14.21.3/SHASUMS256.txt.asc
Downloading node-v14.21.3-win-arm64.7z to compute hashes!
The remote server returned an error: (404) Not Found.
URL https://nodejs.org/dist/v14.21.3/node-v14.21.3-win-arm64.7z is not valid
Could not install [email protected]
Select-CurrentVersion: C:\Users\jbrinkmann\scoop\apps\scoop\current\lib\core.ps1:239
Line |
239 | return $null -ne (Select-CurrentVersion -AppName $app -Global:$gl …
| ~~~~
| Cannot bind argument to parameter 'AppName' because it is an empty string.
InvalidOperation: C:\Users\jbrinkmann\scoop\apps\scoop\current\lib\manifest.ps1:36
Line |
36 | $app = $app.TrimStart('/')
| ~~~~~~~~~~~~~~~~~~~~~~~~~~
| You cannot call a method on a null-valued expression.
Couldn't find manifest for ''.
Possible Solution
As a quick workaround, patching autoupdate.ps1 from
$Manifest.architecture | Get-Member -MemberType NoteProperty | ForEach-Object {
...
}
to
# Arch-spec
$Manifest.architecture | Get-Member -MemberType NoteProperty | Where-Object {
$_.Name -eq (Get-DefaultArchitecture)
} | ForEach-Object {
...
}
But removing the other architectures instead of just not updating them would be cleaner.
System details
Windows version: 10
OS architecture: 64bit
PowerShell version: 7.3.5
Additional software: -
Scoop Configuration
{
"last_update": "2023-07-17T14:09:33.8952617+02:00",
"scoop_branch": "master",
"scoop_repo": "https://github.com/ScoopInstaller/Scoop"
}
Is there at the moment any workaround for the issue or do we have to wait for a fix?
scoop install versions/[email protected].
The feature allows generate the specified version of manifests as much as possible, not being 100% available.
Still experiencing this bug on v0.3.1. The proposed workaround does not match the documentation and should be fixed.
> scoop install "[email protected]"
WARN Given version (16.20.2) does not match manifest (20.5.0)
WARN Attempting to generate manifest for 'nodejs' (16.20.2)
Autoupdating nodejs
Searching hash for node-v16.20.2-win-x86.7z in https://nodejs.org/dist/v16.20.2/SHASUMS256.txt.asc
Found: b0b2ef51565a01e34e4743ab5b26d76d34bb5403479c8b591f5ae434e0ab4adc using Extract Mode
Searching hash for node-v16.20.2-win-x64.7z in https://nodejs.org/dist/v16.20.2/SHASUMS256.txt.asc
Found: ebb0c50b1e4943cfeb005e71f5b9c6733f0a9586df3138df199d855d6abac77c using Extract Mode
Searching hash for node-v16.20.2-win-arm64.7z in https://nodejs.org/dist/v16.20.2/SHASUMS256.txt.asc
Could not find hash in https://nodejs.org/dist/v16.20.2/SHASUMS256.txt.asc
Downloading node-v16.20.2-win-arm64.7z to compute hashes!
The remote server returned an error: (404) Not Found.
URL https://nodejs.org/dist/v16.20.2/node-v16.20.2-win-arm64.7z is not valid
Could not install [email protected]
There is no arm64 option. Worse, specifying -a 64bit or -a 32bit does not change the outcome.
> scoop install -a 64bit "[email protected]"
WARN Given version (16.20.2) does not match manifest (20.5.0)
WARN Attempting to generate manifest for 'nodejs' (16.20.2)
Autoupdating nodejs
Searching hash for node-v16.20.2-win-x86.7z in https://nodejs.org/dist/v16.20.2/SHASUMS256.txt.asc
Found: b0b2ef51565a01e34e4743ab5b26d76d34bb5403479c8b591f5ae434e0ab4adc using Extract Mode
Searching hash for node-v16.20.2-win-x64.7z in https://nodejs.org/dist/v16.20.2/SHASUMS256.txt.asc
Found: ebb0c50b1e4943cfeb005e71f5b9c6733f0a9586df3138df199d855d6abac77c using Extract Mode
Searching hash for node-v16.20.2-win-arm64.7z in https://nodejs.org/dist/v16.20.2/SHASUMS256.txt.asc
Could not find hash in https://nodejs.org/dist/v16.20.2/SHASUMS256.txt.asc
Downloading node-v16.20.2-win-arm64.7z to compute hashes!
The remote server returned an error: (404) Not Found.
URL https://nodejs.org/dist/v16.20.2/node-v16.20.2-win-arm64.7z is not valid
Could not install [email protected]
> scoop install -a 32bit "[email protected]"
WARN Given version (16.20.2) does not match manifest (20.5.0)
WARN Attempting to generate manifest for 'nodejs' (16.20.2)
Autoupdating nodejs
Searching hash for node-v16.20.2-win-x86.7z in https://nodejs.org/dist/v16.20.2/SHASUMS256.txt.asc
Found: b0b2ef51565a01e34e4743ab5b26d76d34bb5403479c8b591f5ae434e0ab4adc using Extract Mode
Searching hash for node-v16.20.2-win-x64.7z in https://nodejs.org/dist/v16.20.2/SHASUMS256.txt.asc
Found: ebb0c50b1e4943cfeb005e71f5b9c6733f0a9586df3138df199d855d6abac77c using Extract Mode
Searching hash for node-v16.20.2-win-arm64.7z in https://nodejs.org/dist/v16.20.2/SHASUMS256.txt.asc
Could not find hash in https://nodejs.org/dist/v16.20.2/SHASUMS256.txt.asc
Downloading node-v16.20.2-win-arm64.7z to compute hashes!
The remote server returned an error: (404) Not Found.
URL https://nodejs.org/dist/v16.20.2/node-v16.20.2-win-arm64.7z is not valid
Could not install [email protected]
Only workaround I can find is to edit the nodejs.json bucket file and remove the arm64 architecture.
@tghw as @HUMORCE suggested, use scoop install versions/[email protected]. You might need to run scoop bucket add versions before, if you don't have it already.
@HUMORCE I agree that the manifest autogeneration cannot be expected to be always working perfectly. I just think that this is a minor issue that actually can be adressed, if considered worth spending time on. Here's an example fix, it's just not very pretty so I didn't consider making a PR.
@JTBrinkmann Thanks, but that does seem like a hacky workaround. In my mind, scoop install XXXX@YYYY should be able to use its version spec as documented without having to add additional buckets. It should also use the -a flag as documented, instead of ignoring it. So, to me, seems like this is still an outstanding bug that needs fixing.
In my mind, scoop install XXXX@YYYY should be able to use its version spec as documented without having to add additional buckets.
Creating separate manifest for major version of app ensures its reliability. As the app undergoes updates, the scripts, persistence, etc. within the manifest may not be applicable to older versions. Because they were changed based on latest version of app.
If you've used Homebrew before, you might think of 'foo' and '[email protected]' as two manifests. In Scoop, <app>@<version> just a method for auto generate manifests for specific versions.
Creating separate manifest for major version of app ensures its reliability. As the app undergoes updates, the scripts, persistence, etc. within the manifest may not be applicable to older versions. Because they were changed based on latest version of app.
If you've used Homebrew before, you might think of 'foo' and '[email protected]' as two manifests. In Scoop,
<app>@<version>just a method for auto generate manifests for specific versions.
If that's the approach Scoop wants to take, I think that's a reasonable discussion to have. But that's not what's currently documented; it's entirely possible with the main bucket to install older versions without adding the versions bucket.
More importantly, that is not the point of this ticket. This ticket is pointing out the incorrect inclusion of arm64 on versions that do not have the associated resource, resulting in 404 errors. Worse, even if arm64 builds were available, scoop is trying to install arm64 on 64bit operating systems, which is also incorrect. This bug occurs even with the architecture is explicitly provided with the -a argument, as I documented above.
But that's not what's currently documented; it's entirely possible with the main bucket to install older versions without adding the versions bucket.
I think this current design is closer to the AUR of Arch Linux. And the documentation mentions that the user needs to add Versions bucket. <app>@<version> is suited for install a recent minor or patch version.
More importantly, that is not the point of this ticket. This ticket is pointing out the incorrect inclusion of arm64 on versions that do not have the associated resource, resulting in 404 errors. Worse, even if arm64 builds were available, scoop is trying to install arm64 on 64bit operating systems, which is also incorrect. This bug occurs even with the architecture is explicitly provided with the -a argument, as I documented above.
Yes, my last reply was just in response to another issue you mentioned.
Improvements to auto-generate function might be:
Generate current architecture(recognized by Scoop Core, fallback allowed) or specified architecture(when -a <...>) only. Then, in an arm64 device:
$arch = current_architecture
if arg.a: $arch = arg.a
if $arch not in manifest.architectures:
$arch = <fallback>
generate...
I think this current design is closer to the AUR of Arch Linux. And the documentation mentions that the user needs to add Versions bucket.
<app>@<version>is suited for install a recent minor or patch version.
That documentation certainly mentions installing the versions bucket, but the title is Switching Ruby, Python and PHP Versions, which not only suggests its use only for specific packages, but also a specific use case. If using the versions bucket is the intended workflow for older versions, that documentation needs to be more obviously placed, both in the top level README and in the --help.
Currently, scoop install --help makes no mention of the versions bucket, instead indicating that the manifest for any version will be auto-generated. I think it's far more likely for someone to check scoop install --help and expect it to work as documented there than to go to the Switching Ruby, Python and PHP Versions wiki page and figure out that applies to old versions of some, but not all, packages in main.
$ scoop install --help
Usage: scoop install <app> [options]
e.g. The usual way to install an app (uses your local 'buckets'):
scoop install git
To install a different version of the app
(note that this will auto-generate the manifest using current version):
scoop install [email protected]
...
Oh, the help text not be clear enought, I think an instruction should be added about Versions bucket at same time.
Added a new ticket (#5783) requesting documentation as discussed. For anyone scrolling to the end of this ticket, I believe the issue is still outstanding in the misidentification of architecture between arm64 and 64bit, as documented earlier in this ticket.
> scoop bucket list
Name Source Updated Manifests
---- ------ ------- ---------
extras https://github.com/ScoopInstaller/Extras 2024-01-10 오후 1:26:52 1892
main ~\scoop\buckets\main 2024-01-10 오후 1:29:19 1268
versions https://github.com/ScoopInstaller/Versions 2024-01-10 오후 1:32:36 435
> scoop install [email protected]
WARN Given version (3.9.13) does not match manifest (3.12.1)
WARN Attempting to generate manifest for 'python' (3.9.13)
Autoupdating python
Searching hash for python-3.9.13.exe in https://www.python.org/downloads/release/python-3913/
Found: md5:46c35b0a2a4325c275b2ed3187b08ac4 using Extract Mode
Searching hash for python-3.9.13-amd64.exe in https://www.python.org/downloads/release/python-3913/
Found: md5:e7062b85c3624af82079794729618eca using Extract Mode
Searching hash for python-3.9.13-arm64.exe in https://www.python.org/downloads/release/python-3913/
Could not find hash in https://www.python.org/downloads/release/python-3913/
Downloading python-3.9.13-arm64.exe to compute hashes!
The remote server returned an error: (404) Not Found.
URL https://www.python.org/ftp/python/3.9.13/python-3.9.13-arm64.exe#/setup.exe is not valid
Could not install [email protected]
Select-CurrentVersion: C:\Users\SSAFY\scoop\apps\scoop\current\lib\core.ps1:239
Line |
239 | return $null -ne (Select-CurrentVersion -AppName $app -Global:$gl …
| ~~~~
| Cannot bind argument to parameter 'AppName' because it is an empty string.
InvalidOperation: C:\Users\SSAFY\scoop\apps\scoop\current\lib\manifest.ps1:36
Line |
36 | $app = $app.TrimStart('/')
| ~~~~~~~~~~~~~~~~~~~~~~~~~~
| You cannot call a method on a null-valued expression.
Couldn't find manifest for ''.
> scoop install versions/[email protected]
WARN Given version (3.9.13) does not match manifest ()
WARN Attempting to generate manifest for 'python' (3.9.13)
'python' does not have autoupdate capability
couldn't find manifest for '[email protected]'
Python also tries to download arm64 package on x64 device.
Pythonalso tries to download arm64 package on x64 device.
This is because Scoop attempts to generate manifest for python 3.9.13 based on 'main/python,' and the manifest defines that includes the arm64 architecture, but the upstream does not provide relevant resources.
For python 3.9.13, you should install versions/python39:
scoop bucket add versions
scoop install python39
Explanation about 'Versioned manifest': https://github.com/ScoopInstaller/Scoop/issues/5582#issuecomment-1883426601
This is because Scoop attempts to generate manifest for python 3.9.13 based on 'main/python,' and the manifest defines that includes the arm64 architecture, but the upstream does not provide relevant resources.
I still think this is missing the underlying issue in this ticket. On a 64bit machine, there is no reason Scoop should be trying to download the package for arm64, whether you got it from main with the generated manifest or from versions. arm64 is not the correct architecture for the machine installing it.
The comment from @TheNoFace shows that they had versions installed, and when they used it, a manifest could not be found. My guess is that's because it was using an arm64 manifest from versions (which does not exist) and not 64bit as it should have.
Maybe the title of this ticket is misleading, but the core issue is that, for some reason, Scoop is determining the incorrect architecture and trying to use it when it shouldn't. That is the bug that needs to be fixed.
I still think this is missing the underlying issue in this ticket. On a 64bit machine, there is no reason Scoop should be trying to download the package for arm64, whether you got it from main with the generated manifest or from versions. arm64 is not the correct architecture for the machine installing it.
How Scoop generate manifest currently:
Force update manifest with specified version, it will consider all architectures defined in autoupdate of manifests. if autoupdate.<architecture>.hash is NOT defined, the resources will be downloaded for hash value.
The comment from @TheNoFace shows that they had versions installed, and when they used it, a manifest could not be found. My guess is that's because it was using an arm64 manifest from versions (which does not exist) and not 64bit as it should have.
-
scoop install [email protected]:main/pythondefined autoupdate.arm64, but upstream does not provides arm64 binary installer for python 3.9.13 -
scoop install versions/[email protected]:versions/[email protected]is wrong manifest name, becauseversions/pythondoes not exists. instead,versions/python39current version is 3.9.13, so we does not need specify a version when installing it. but you can try installversions/[email protected].
Maybe the title of this ticket is misleading, but the core issue is that, for some reason, Scoop is determining the incorrect architecture and trying to use it when it shouldn't. That is the bug that needs to be fixed.
In fact, the feature is working as expected for current design, but you can't understand why resources of other architectures has been downloaded when generating a manifest. And it already responsed in https://github.com/ScoopInstaller/Scoop/issues/5582#issuecomment-1883504486 and Possible Solution by @JTBrinkmann .
Yes, I'm understanding better now how it's working, though I can't say I understand why anyone would want to download all architectures, as it seems that would hurt performance. Clearly that was a decision made at some point, it's just not clear why.
I still think that a fix is applicable here. The versions bucket is poorly documented, and even if it was well documented, the truth is most people don't RTFM. If scoop failed gracefully on a missing manifest and still tried to install the appropriate architecture, erroring out only if it can't find that specific architecture, that would address this issue.
Clearly work went into the auto-generated manifests. It seems silly to void all that work with this brittleness.
Clearly that was a decision made at some point, it's just not clear why.
This PR has a complete record: https://github.com/ScoopInstaller/Scoop/pull/1332
Okay, I'm coming back to this issue after encountering it for the second time. Both times the install would have succeeded were it not for trying to download an ARM version of a library that didn't ship ARM until much later. I cannot understand why this change faces so much opposition given how simple & straightforward it is. Instead of downloading all binaries for all architectures you simply reduce it to only the architecture that the user currently uses (I mean realistically its only going to be X64 & ARM). That way if the binary fails to download there is a genuine reason why Scoop cannot install the package. And if it succeeds then scoop can still install an older version without requiring a whole extra bucket just by ignoring the architectures that it didn't need in the first place.
This change seems fairly straightforward and while it will modify the behavior of auto generating old manifests and makes it a little more complex it does speed it up by only downloading the architecture necessary. An absolutely worthwhile tradeoff making Scoop just that much more usable and intuitive. Since the only time installing an old version should fail if there is a genuinely breaking change for the current architecture between the current and desired version.
Happy to contribute to make this happen 👍🏻
I agree with @CEbbinghaus on this ticket. It makes no sense to slow down installation and increase the likelihood of failure to download all architectures on install. The behavior should be changed to either:
- Accept failures while downloading the architecture, and let it fail on install if the necessary architecture is missing. This is probably the easier option, as the failure on install will likely already happen, and only requires passing on download errors.
- Stop downloading other architectures unnecessarily. This is likely a more intensive change, but would make the tool as a whole faster and more robust.
Either way, the current feedback of "Use an undocumented bucket" is a bad kludge and is actively causing people difficulty.
