pdm

pdm copied to clipboard

Is your feature request related to a problem? Please describe.

I need to define multiple pypi mirrors. As far as I understand it, using pdm.config pypi.url, I can set ONE mirror.

Describe the solution you'd like

There should be a way to define multiple pypi mirrors.

Alternatively, when adding a package, there should be a parameter available that specifies the mirror that will be used to look for the package.

Hi, did you read https://pdm.fming.dev/latest/pyproject/tool-pdm/#specify-other-sources-for-finding-packages? Does that work for you?

What I don't like about this solution is the fact I don't want to reveal any internal server names in my pyproject.toml file. If I do that, committing to VCS doesn't make sense anymore, in my view. If somebody from outside decides to fork the repo, he has some server in the pyproject.toml that doesn't make any sense to him. He won't be able to access our internal servers anyway. And maybe he needs to define his own internal servers.

To cut a long story short, I think setting multiple pypi mirrors in pdm's config makes much more sense. It's a private configuration, so the details shouldn't be put into a publicy available file.

I agree. I'm lucky enough to have a virtual repository that maps to several others, so I can just set one pypi url, but if that wasn't the case I'd be asking for the same thing. Setting the repositories in pyproject.toml is not an acceptable workaround in my case, and it's actually one of the reason I moved away from Poetry and tried PDM :smile:

@andreas-vester As you said the mirror URLs are private settings, so all packages are supposed to be able to install if others don't have such private settings. Is that right?

Well yes, this is right.

However, my private settings (i.e. server names) are of no use to other users. Besides, I don't want to reveal any internal names in pyproject.toml for others to see. As @pawamoy pointed out, this is the same behavior as can be seen with poetry.

When using pip, those settings are stored in pip.ini, where I can specify index-url or extra-index-url. As far as I understand it, pdm is using pip under the hood, isn't it? Would it be possible to use pip.ini's settings as a workaround?

A better/other solution could be to have dedicated pdm settings, something like

• pypi.url1
• pypi.url2
• pypi.url3
• ...

What do you think?

A better/other solution could be to have dedicated pdm settings, something like

• pypi.url1
• pypi.url2
• pypi.url3
• ...

What do you think?

I agree that there is scenario for multiple private index sources, but I am struggling with how to make the UX better. The above proposal doesn't look good to me.

As far as I understand it, pdm is using pip under the hood, isn't it? Would it be possible to use pip.ini's settings as a workaround?

No, PDM no longer uses pip under the hood.

OK, agreeing on putting the settings outside the pyproject.toml is a very good step. This would clearly differentiate pdm from poetry.

I'm not an expert, but another possibility (other than pdm config) could potentially be working with environment variables. You're already using environment variables in the dependency specification. So why not taking all information from these variables?

Let's say you have a .env file with some variables like

• PRIVATE_PYPI_URL_SERVER1="https://some.private.pypi.mirror.com"
• PRIVATE_PYPI_URL_SERVER2="https://another.private.pypi.mirror.com"

You could identify the relevant variables with the prefix PRIVATE_PYPI_URL.

Just a very first suggestion...

@andreas-vester One option might be to:

• Define mirrors in pyproject.toml with paths like https://source.private
• Modify your /etc/hosts to map the fake domain source.private to the appropriate IP address

...this shifts responsibility from pdm to your development environment, and you can safely store the fake domain in version control. It even allows you to define different sources in development and production (if you want). If you're working on a team, you might want to provide everyone with a script that does the job.

Since PDM was previously reading pip's config, why not supporting the same kind of settings: a main index and a list of extra ones? Or, since all indexes are checked anyway, support setting pypi.url both as a string (single index) or a list of strings (multiple indexes)?

Single index

pdm config pypi.url https://singleindex

[pypi]
url = "https://singleindex"

Multiple indexes

pdm config pypi.url https://index1 https://index2

[pypi]
url = [
    "https://index1",
    "https://index2",
]

I've read all the proposals but none is satisfying.

@andreas-vester introduced .env files to configure pdm, which I strongly -1. Users are already confused by project metadata(pyproject.toml) and system/user/project settings. .env file is only a replacement for environment variables used as temporary modifiers, rather than configurations of any kind.

@lofidevops offers a workaround to solve the described problem at present.

@pawamoy Your proposal looks almost good but remember we also support pypi.verify_ssl(boolean) config item, and ideally, it should be set for each index instead of all. As a result, this solution becomes less useful.

Although I admit the usefulness of this feature, I can't come up with a good UI to read and write the settings. More ideas are welcome.

---

UPDATE: based on @pawamoy 's proposal, maybe we can deprecate the pypi.verify_ssl config and derive it from the URL scheme(verify_ssl = true for https and false otherwise), but that will bring a side-effect that users can't disable the SSL verification against an HTTPS index.

Ah, indeed, I didn't think about other related settings. Supporting multiple indexes would then require a more drastic change I guess, something like:

[pypi.private-server-1]
url = "https://index1"
verify_ssl = true

[pypi.private-server-2]
url = "https://index2"
verify_ssl = false

i also have the case where this would be needed, eagerly awaiting a solution.

Same here, for projects using pypi + a company private repository.

Please be noted this feature only applies to those projects that can be installed successfully even if other people change the local PyPI URL(private config).

That is to say, if your project depends on some packages that are only available on a private index, making the source not interchangeable, just use the tool.pdm.source array in pyproject.toml(shared config).

One option might be to:

* Define mirrors in pyproject.toml with paths like `https://source.private`

* Modify your `/etc/hosts` to map the fake domain `source.private` to the appropriate IP address

...this shifts responsibility from pdm to your development environment, and you can safely store the fake domain in version control. It even allows you to define different sources in development and production (if you want). If you're working on a team, you might want to provide everyone with a script that does the job.

@lofidevops I don't want to put any internal (or fake) server names in pyproject.toml. If I commit this to GitHub and somebody clones the repo, those server names wouldn't be of any use for him.

Or did I miss your point??

Please be noted this feature only applies to those projects that can be installed successfully even if other people change the local PyPI URL(private config).

That is to say, if your project depends on some packages that are only available on a private index, making the source not interchangeable, just use the tool.pdm.source array in pyproject.toml(shared config).

I think it is clear that if you use some libraries that are exclusively available to you (via a private repo), that publishing this project and expect others to be able to successfully clone it is not possible.

However, my issue is that we use a pypi mirror in our company. That is, I am looking for a way to define all dependencies in pyproject.toml, but specifiying the source server should be done in a private (configuration) file. This way, I can install from my private mirror, while people from outside the company simply use the standard pypi source.

@pawamoy I found your blog entry somewhat-modern-python-development, where you mentioned the following:

Then I learned about PDM. It had all the good things Poetry had, and it removed the previously mentioned pain points, namely: it was able to read pip's configuration. It made using private indexes so easy: setup pip's index-url and trusted-host and you're done. Your own configuration does not contaminate other users through pyproject.toml (note: PDM 2.0 doesn't read pip's configuration anymore, but you can still configure your index outside of pyproject.toml).

The very last sentence drew my particular attention and I am wondering what you exactly mean? How you do configure your index outside pyproject.toml?

@andreas-vester PDM currently allows to set exactly one URL for the default PyPI-like index to use: pdm config pypi.url https://.... This config lands in ~/.config/pdm/config.toml. That's what I meant in the blog post :)

I think it is clear that if you use some libraries that are exclusively available to you (via a private repo), that publishing this project and expect others to be able to successfully clone it is not possible.

@andreas-vester Chances are you may also share the project with someone inside your company, or deploy it to another machine, right? If you depend on a private library but don't specify the source in pyproject.toml, you push all others with the same local config, otherwise, the project can't be installed on their machines.

Currently, PDM supports two kinds of package sources:

• Local source(single, defined in $PDM_CONFIG/config.toml)
• Shared sources(multiple, defined in tool.pdm.source in pyproject.toml)

The local source can be overridden by the shared source with pypi key.

One should notice that multiple local sources will make the situation complicated and I haven't figured out a clean way to make them coexist.

• What will be overridden by the pypi shared source after the change?
• Is it possible to override the other local sources? If yes, how? For example, local and shared both contain three sources, what is the final list of source URLs to find packages from? Will there be any deduplication?
• the pdm config CLI

I don't want to put any internal (or fake) server names in pyproject.toml. If I commit this to GitHub and somebody clones the repo, those server names wouldn't be of any use for him.

Or did I miss your point??

@andreas-vester You didn't miss my point, I was assuming only private team members had access to your repo. So my proposal doesn't fit your use-case. Sorry about that! But it looks like the subsequent comments are zeroing in a solution. Good luck!

Our organisation would really appreciate this feature, since we have a build server which needs to talk to multiple private repos which requires authentication (so we can't store user/pass in the pyproject.toml). We were using the extra-index-url of pip when using PDM v1 previously...

What will be overridden by the pypi shared source after the change? Is it possible to override the other local sources? If yes, how? For example, local and shared both contain three sources, what is the final list of source URLs to find packages from? Will there be any deduplication?

With regards to the above would it be possible to have some sort of hierarchy to derive the list and order of sources? I.e. read from pyproject.toml first (most specific) then config.toml (least specific) And then concatenate the list of pypi.indexes with their own urls ?

(so we can't store user/pass in the pyproject.toml).

If it is the only concern you can use environment variables

I have a similar need, but mainly with authentication and not with server names. The problem with environment variables I have is that while it works great in CI, when installing locally it requires meddling with your shell configuration and it's not very portable (the variable names may conflict for different users and repos).

Ideally I'd like to have user-wide config https://github.com/pdm-project/pdm/issues/1310#issuecomment-1216670288 where I can store the auth part or have the source urls in pyproject but make auth work with a keyring (like artifacts-keyring) and also with env variables or in the worst case enable dotenv support for installs.

What will be overridden by the pypi shared source after the change?

I'd guess that if someone wants to disable the default index then it's because the shared source config is self contained, and the local config will not be taken into account, however the way it's configured may not make much sense now. Maybe add an item under tool.pdm.resolution that sets this behavior?

Is it possible to override the other local sources? If yes, how? For example, local and shared both contain three sources, what is the final list of source URLs to find packages from? Will there be any deduplication?

Do similar thing to what pip is doing when reading from various config files would be my guess (i.e. just concat?), but prefer shared sources first when respect-source-order.

the pdm config CLI

The config may support both pypi.url and the new pypi.<name>.url if you want to keep backward compatibility. Then there will be a distinction similar to index-url / extra-index-url between overriding the default index via pypi.url and supplying additional ones via pypi.<name>.url.

Wow, great work @frostming!

@frostming This is just awesome! THANK YOU!

It now allows me to use our internal pypi indexes and corresponding credentials while not being forced to put them into pyproject.toml. This will allow me to put it into GitHub and share it with the public.

This feature enables me to make the move to pdm!!

In my view, this is a unique selling point and one of the major differences compared to poetry. You should go and advertise this feature.

pdm deserves more appreciation within the wider Python community!

Hey, read the whole conversation, love the feature. But still didn't understood how to store the username/password/credentials outside of the pyproject.toml.

PS: I Love pdm

Hey, read the whole conversation, love the feature. But still didn't understood how to store the username/password/credentials outside of the pyproject.toml.

PS: I Love pdm

Found the solution I wanted looking in the docs.

This is my solution to the config for future viewers of this issue. PROJECT/pyproject.toml

[tool.pdm.resolution]
respect-source-order = true

[[tool.pdm.source]]
name = "NAME_1"
url = "URL_1"

[[tool.pdm.source]]
name = "NAME_2"
url = "URL_2"

$PDM_CONFIG/config.toml

[pypi.NAME_1]
username = "USERNAME_1"
password = "PASSWORD_1"

[pypi.NAME_2]
username = "USERNAME_2"
password = "PASSWORD_2"

$PDM_CONFIG for macOS is in ~/Library/Preferences/pdm/config.toml