fix: Custom Datasource with `file://` not working

[balihb]

I'm creating this PR cause I wanted to update to the latest Web App Stack for Python Azure Functions.

Seems like Custom Datasource won't use hostRules. Neither there is any way to configure OAuth2, which is problematic since Microsoft only hands out tokens for one hour and updating multiple repositories can take longer.

After that I checked in the latest Web App Stack list Json file into the repository, but Renovate won't read it either and it will only give "issues":{"releases":"Required"}},"msg":"Response has failed validation" instead of any reasonable error.

I've modified the code to give some more reasonable debug messages, but for some reason, it won't find the file in the project. The only case when I get a meaningful debug message is if I put the file outside of the repository path.

Am I missing something here or using a file:// URL is just not working?

{"name":"renovate","hostname":"***","pid":123,"level":20,"logContext":"***","repository":"***","localPath":"/builds/***/renovate/repos/gitlab/***/azurePythonWebAppStacks.json","msg":"Local file not found","time":"2024-02-12T22:08:59.750Z","v":0}
{"name":"renovate","hostname":"***","pid":123,"level":20,"logContext":"***","repository":"***","parentDirList":{},"msg":"Parent dir list","time":"2024-02-12T22:08:59.750Z","v":0}
{"name":"renovate","hostname":"***","pid":123,"level":20,"logContext":"***","repository":"***","err":{"message":"Schema error","stack":"ZodError: Schema error\n    at Object.get error [as error] (/usr/local/renovate/node_modules/.pnpm/[email protected]/node_modules/zod/lib/types.js:43:31)\n    at ZodObject.parse (/usr/local/renovate/node_modules/.pnpm/[email protected]/node_modules/zod/lib/types.js:143:22)\n    at CustomDatasource.getReleases (/usr/local/renovate/lib/modules/datasource/custom/index.ts:63:45)\n    at fetchReleases (/usr/local/renovate/lib/modules/datasource/index.ts:296:13)\n    at lookupUpdates (/usr/local/renovate/lib/workers/repository/process/lookup/index.ts:115:56)\n    at withLookupStats (/usr/local/renovate/lib/workers/repository/process/fetch.ts:27:18)\n    at fetchDepUpdates (/usr/local/renovate/lib/workers/repository/process/fetch.ts:72:30)\n    at /usr/local/renovate/node_modules/.pnpm/[email protected]/node_modules/p-map/index.js:57:22","issues":{"releases":"Required"}},"msg":"Response has failed validation","time":"2024-02-12T22:08:59.771Z","v":0}
{"name":"renovate","hostname":"***","pid":123,"level":20,"logContext":"***","repository":"***","dependency":"azureWebAppPython","packageFile":"azure/functionApp.bicep","msg":"Failed to look up custom.azureWebAppPython package azureWebAppPython","time":"2024-02-12T22:08:59.772Z","v":0}

The Custom Datasource looks like this:

{
  "customDatasources": {
    "azureWebAppPython": {
      "defaultRegistryUrlTemplate": "file://azurePythonWebAppStacks.json",
      "transformTemplates": [
        "{\"releases\": $$.value[name=\"python\"].properties.majorVersions[value=\"3\"].minorVersions.({\"version\": $.value, \"isDeprecated\": $.stackSettings.linuxRuntimeSettings.isDeprecated})}"
      ]
    }
  },
  "description": "Get Python versions from `https://learn.microsoft.com/en-us/rest/api/appservice/provider/get-web-app-stacks?view=rest-appservice-2022-03-01&tabs=HTTP`. Require Azure credentials."
}

The Regex Manager:

{
  "customManagers": [
    {
      "customType": "regex",
      "datasourceTemplate": "custom.azureWebAppPython",
      "depNameTemplate": "azureWebAppPython",
      "fileMatch": [
        "\\.bicep$"
      ],
      "matchStrings": [
        "(?:\\/\\/\\s+renovate:(?: datasource=(?<datasource>[a-z-.]+?))?(?: depName=(?<depName>[^\\s]+?))?(?: packageName=(?<packageName>[^\\s]+?))?(?: versioning=(?<versioning>[a-z-0-9]+?))?(?: extractVersion=(?<extractVersion>[^\\s]+?))?)?\\s+linuxFxVersion: 'PYTHON\\|(?<currentValue>[0-9]\\.[0-9]+)'"
      ]
    }
  ],
  "description": "Update `linuxFxVersion` Python variables in `.bicep` files."
}

[CLAassistant]

All committers have signed the CLA.

[balihb]

The documentation for the provider is here:

https://learn.microsoft.com/en-us/rest/api/appservice/provider/get-web-app-stacks?view=rest-appservice-2022-03-01&tabs=HTTP

but it is kind of irrelevant for the current context of the PR.

[balihb]

What's the expected behavior after this change if the user either:

* Specifies an absolute (and non-local) path? e.g. `/etc/passwd`

* Specifies a relative (and non-local) path? (e.g. `../../../etc/password`)

in that case everything stay the same as before.

[balihb]

my problem is, that it looks like the localDir is empty (not populated) when the custom datasource is evaluated.

[rarkins]

Are you sure this won't allow an attacker to list the contents of any folder in the file system due to the new log message?

[balihb]

Are you sure this won't allow an attacker to list the contents of any folder in the file system due to the new log message?

removed dir listing

I've tried to reproduce the problem with no luck:

https://github.com/balihb/renovate-test

[rarkins]

Are you sure this won't allow an attacker to list the contents of any folder in the file system due to the new log message?

removed dir listing

I've tried to reproduce the problem with no luck:

balihb/renovate-test

Reproduce which problem exactly?

[balihb]

Reproduce which problem exactly?

two problems on the private gitlab repository:

• custom datasource won't pick up the token for management.azure.com and will give 403.
• if I pre download the data and check it into the repository (as in the github repo), the file is seemingly not there and listing the localDir returns an empty list.

[balihb]

two problems on the private gitlab repository:

* custom datasource won't pick up the token for `management.azure.com` and will give 403.

* if I pre download the data and check it into the repository (as in the github repo), the file is seemingly not there and listing the localDir returns an empty list.

the auth problem is present on github too:

https://github.com/balihb/renovate-test/actions/runs/7902953740/job/21569751381#step:4:360

[balihb]

the auth problem is present on github too:

https://github.com/balihb/renovate-test/actions/runs/7902953740/job/21569751381#step:4:360

still after a few modification:

https://github.com/balihb/renovate-test/actions/runs/7914008705/job/21602842187#step:5:367

[viceice]

the datasource file needs to be inside the processed repository, not inside the runner repo.

that's probably your issue over all.

your simplest solution is to host your jsonc file on a simple webserver, so you can reuse it on multiple repos.

[github-actions[bot]]

Please create a GitHub Discussion instead of this issue.

Issues in this repository are for creation by Maintainers only - please create a GitHub Discussion instead. If needed, a Renovate maintainer will create an Issue after your Discussion been triaged and confirmed.

This Issue will now be closed and locked. We may later batch-delete this issue. This way we keep Issues actionable, and free of duplicates or wrong bug reports.

Thanks, the Renovate team

[balihb]

the datasource file needs to be inside the processed repository, not inside the runner repo.

that's probably your issue over all.

your simplest solution is to host your jsonc file on a simple webserver, so you can reuse it on multiple repos.

the file was in the repo root.

[rarkins]

@balihb if you can create a reproduction repo then we'll take a look