Complete PIP_INDEX_URL as environment variable isn't parsed in [[tool.pdm.source]]

[vbennich]

• [x] I have searched the issue tracker and believe that this is not a duplicate.

Make sure you run commands with -v flag before pasting the output.

Steps to reproduce

When trying to work with a private python feed, It would be nice to get the environment variable PIP_INDEX_URL and pass it straight to pdm. As it is today I think we need to supply at least a "@" in the url for it to be parsed correctly. Although my PIP_INDEX_URL is already contains the https://PAT@myOrg/myFeed PIP_INDEX_URL is also generated in azure pipelines which would make it neat to directly use that in the pyproject.toml.

pyproject.toml:

[[tool.pdm.source]]
url = "${PIP_INDEX_URL}"
verify_ssl = false
name = "pypi"
ca_certs = "${PIP_CERT}"

pdm install

Actual behavior

requests.exceptions.MissingSchema: Invalid URL '${PIP_INDEX_URL}/mypackage': No scheme supplied. Perhaps you meant https://${PIP_INDEX_URL}/mypackage/?

Expected behavior

The url = "${PIP_INDEX_URL}" is parsed.

Environment Information

# Paste the output of `pdm info && pdm info --env` below:
PDM version:
  2.9.3
Python Interpreter:
  /home/user/Documents/myProj/.venv/bin/python (3.8)
Project Root:
  /home/user/Documents/myProj/
Local Packages:
  
{
  "implementation_name": "cpython",
  "implementation_version": "3.8.10",
  "os_name": "posix",
  "platform_machine": "x86_64",
  "platform_release": "5.15.0-83-generic",
  "platform_system": "Linux",
  "platform_version": "#92~20.04.1-Ubuntu SMP Mon Aug 21 14:00:49 UTC 2023",
  "python_full_version": "3.8.10",
  "platform_python_implementation": "CPython",
  "python_version": "3.8",
  "sys_platform": "linux"
}

[vbennich]

This could be relevant: https://github.com/pdm-project/pdm/issues/997

[frostming]

You may think expanding env vars is as straightforward as a simple call of os.expandvars() and surprised at why PDM still doesn't support this! but that is not. When expanding env vars in URL, you must keep in mind of the URL encoding, which may cause unexpected behavior. That's why we only support expanding vars in auth part now.

Specifically, if you have a env var named FOO and the value contains a special character.

• When it's in the auth part like https://${FOO}:@example.com, it should be encoded when expanded
• When it's in the path part, like https://example.com/${FOO}, it should be encoded
• But when it is the whole URL, should it be encoded or not? If yes, it is against the user's intuitive in how to build a URL; If no, this is an inconsistency.

So in short, to support arbitrary env var expansion, we need more work to make the UI intuitive and consistent so that users don't have to bother with too many hidden rules. But that seems unresolvable now.

[sanmai-NL]

Why not leave the responsibility for encoding and decoding to the caller?

[frostming]

Do you usually encode chars when you store a password in env var?

What if the env var is also used by other programs?