Package Internalizer - Cannot Replace Environment Variables in URL Strings

[ryanrichter94]

Issue

Cannot internalize a package that has an environment variable in the download URL path

Example

$url = "https://s3.amazonaws.com/ddagent-windows-stable/ddagent-cli-$($env:chocolateyPackageVersion).msi"

Choco Error Response

Unable to find value for variable '$($env'. May not be able to download resource from url 'https://s3.amazonaws.com/ddagent-windows-stable/ddagent-cli-$($env:chocolateyPackageVersion).msi'.

Download doesn't yet support replacement of string methods. May not be able to download resource from url 'https://s3.amazonaws.com/ddagent-windows-stable/ddagent-cli-$($env:chocolateyPackageVersion).msi'.

References

• ZenDesk Ticket

┆Issue is synchronized with this Gitlab issue by Unito

[ferventcoder]

How it works when it is not an environment variable:

[shawnfunke]

We re-compile packages for internal usages, but fail for the prometheus-windows-exporter.install package since it contains environment variables. We'd like to prevent to fork and change affected packages whenever possible. For us it's a lot easier to just whitelist specific package versions and mirror them directly instead of building them from scratch. @ferventcoder, is there anything new regarding this issue?

$ErrorActionPreference = 'Stop';

$packageName= 'prometheus-windows-exporter.install'
$toolsDir   = "$(Split-Path -parent $MyInvocation.MyCommand.Definition)"
$version    = "$($env:chocolateyPackageVersion)"
$url        = "https://github.com/prometheus-community/windows_exporter/releases/download/v$version/windows_exporter-$version-386.msi"
$url64      = "https://github.com/prometheus-community/windows_exporter/releases/download/v$version/windows_exporter-$version-amd64.msi"

https://github.com/andrewmostello/prometheus-windows-exporter.install/blob/8a3b05a722b9251c3e8cca3caea46d51403a8ee4/tools/chocolateyinstall.ps1#LL1C1-L7C137

prometheus-windows-exporter.install v0.22.0 (forced) [Approved]
Found internalizable Chocolatey functions. Inspecting values for remote resources.
Download doesn't yet support replacement of string methods. May not be able to download resource from url 'https://github.com/prometheus-community/windows_exporter/releases/download/v$($env:chocolateyPackageVersion)/windows_exporter-$($env:chocolateyPackageVersion)-386.msi'.
Found 'https://github.com/prometheus-community/windows_exporter/releases/download/v$($env:chocolateyPackageVersion)/windows_exporter-$($env:chocolateyPackageVersion)-386.msi' for replacement.
Download doesn't yet support replacement of string methods. May not be able to download resource from url 'https://github.com/prometheus-community/windows_exporter/releases/download/v$($env:chocolateyPackageVersion)/windows_exporter-$($env:chocolateyPackageVersion)-amd64.msi'.
Found 'https://github.com/prometheus-community/windows_exporter/releases/download/v$($env:chocolateyPackageVersion)/windows_exporter-$($env:chocolateyPackageVersion)-amd64.msi' for replacement.
Encountered error downloading package prometheus-windows-exporter.install v:
System.Exception: The remote file either doesn't exist, is unauthorized, or is forbidden for url 'https://github.com/prometheus-community/windows_exporter/releases/download/v$($env:chocolateyPackageVersion)/windows_exporter-$($env:chocolateyPackageVersion)-386.msi'. 
  The remote server returned an error: (404) Not Found.
   at chocolatey.licensed.infrastructure.app.commandresources.RemoteDownloader.download_file(String url, String filePath, String userAgent, Boolean showProgress)
   at chocolatey.licensed.infrastructure.app.commands.ProChocolateyDownloadCommand.process_downloaded_package(IPackage package, String downloadLocation, ChocolateyConfiguration configuration)
   at chocolatey.infrastructure.app.nuget.NugetCommon.<>c__DisplayClass14.<GetPackageManager>b__11(Object s, PackageOperationEventArgs e)
   at NuGet.PackageManager.OnInstalled(PackageOperationEventArgs e)
   at NuGet.PackageManager.Execute(IPackage package, IPackageOperationResolver resolver)
   at NuGet.PackageManager.InstallPackage(IPackage package, Boolean ignoreDependencies, Boolean allowPrereleaseVersions)
   at chocolatey.licensed.infrastructure.app.commands.ProChocolateyDownloadCommand.download_packages(String packageName, String packageVersion, String downloadLocation, ChocolateyConfiguration configuration)

[pauby]

@shawnfunke The issue has not been assigned to a milestone, so it's not something we are working on just now.

[shawnfunke]

@pauby is there any chance of it being addresses in the near future?

[pauby]

@shawnfunke We do have another release of Chocolatey Licensed Extension planned this year (after the upcoming 6.0.0 release). But even with the best will in the world, plans do change.

I'm not sure that helps and I apologise for that.

[shawnfunke]

@pauby alright, thanks for clarifying!

[Stunkymonkey]

@shawnfunke a current workaround for me: choco download prometheus-windows-exporter.install --no-progress --version 0.22.0 --internalize-all-urls --internalize version=0.22.0

[shawnfunke]

@Stunkymonkey what you mentioned isn't a workaround, there's been a fix regarding the package version substitution in the released version 6.1.0 ^1. After a quick test I can confirm that this has solved the above mentioned issue.

@pauby I'm not sure if there is anything else regarding this issue, if not I'd be fine with marking this issue as resolved.

[pauby]

@shawnfunke I don't think this has been fixed as it's not in the 6.1.0 milestone and therefore completed. Just testing it on a dummy package, with Chocolatey Licensed Extension 6.1.0, I get:

Chocolatey v2.1.0 Business
Downloading existing package(s) to C:\tmp\snagit\snagit\internal\download

snagit v2022.1.2.20221010 (forced)
Found internalizable Chocolatey functions. Inspecting values for remote resources.
Unable to find value for variable '$($env'. May not be able to download resource from url 'https://download.techsmith.com/snagit/releases/$($env:ChocolateyPackageVersion)/snagit.msi'.
Download doesn't yet support replacement of string methods. May not be able to download resource from url 'https://download.techsmith.com/snagit/releases/$($env:ChocolateyPackageVersion)/snagit.msi'.
Found 'https://download.techsmith.com/snagit/releases/$($env:ChocolateyPackageVersion)/snagit.msi' for replacement.

[OraDotNetDev]

I was wondering whether the Internalizer's processing of the ChromeDriver community package is also covered by this issue?

When the Internalizer executes against ChromeDriver, the Json manifest (listing installation payloads) referenced within chocolateyinstall.ps1 is captured and added to the recompiled package, however not the target payload/archive ie https://edgedl.me.gvt1.com/edgedl/chrome/chrome-for-testing/116.0.5845.96/win32/chrome-win32.zip (see output below.)

Thanks

OraDotNetDev

# chromedriver
# chocolateyinstall.ps1

$jsonUrl = "https://googlechromelabs.github.io/chrome-for-testing/known-good-versions-with-downloads.json"
# Fetch the JSON content from the URL
$jsonContent = Invoke-RestMethod -Uri $jsonUrl
# Find the desired versioninfo
$desiredVersionInfo = $jsonContent.versions | Where-Object { $_.version -eq $chromedriverVersion }

# Find the Chromedriver download URL for the desired platform
$chromedriverUrl = $desiredVersionInfo.downloads.$packageName | 
    Where-Object { $_.platform -eq $desiredPlatform } | 
    Select-Object -ExpandProperty url

Install-ChocolateyZipPackage "packageName" -url "$chromedriverUrl" -unzipLocation "$toolsDir" -checksumType "sha256" -checksum "$checkSum"

PS C:\> choco download chromedriver --source='C:\Temp\Source' --internalize -i
Chocolatey v2.2.2 Business
Downloading existing package(s) to C:\download

chromedriver v116.0.5845.96
Found internalizable Chocolatey functions. Inspecting values for remote resources.
Found 'https://googlechromelabs.github.io/chrome-for-testing/known-good-versions-with-downloads.json' for replacement.
Downloading: 100% - 349.28 KB
Download of 'known-good-versions-with-downloads.json' completed.
Replacing 'https://googlechromelabs.github.io/chrome-for-testing/known-good-versions-with-downloads.json' with '$(Split-Path -parent $MyInvocation.MyCommand.Definition)\files\known-good-versions-with-downloads.json'

Adding -UseOriginalLocation to Install-ChocolateyPackage (if found).
 You can also turn off the feature 'internalizeAppendUseOriginalLocation' to not append
Updating chocolateyInstall.ps1 with local resources.
Recompiling package...

[pauby]

@OraDotNetDev While fixing this issue may fix the internalization of that package, we don't support any packages on the Chocolatey Community Repository. If the package can be amended to work with the current internalization, it would be best to get it amended by contacting the maintainer directly.