Request for `uv.lock` to support different index urls across different developer machines and CI environments

[humanzz]

Hello team,

Many thanks for the awesome work you're doing, and the countless amounts of time you've saved everyone using uv (and ruff).

I've seen the recent blog post about 0.3.0 and the accompanying documentation update, and from playing a bit to test commands like uv lock and uv sync, I actually have a question about uv.lock.

I've described the setup I use in https://github.com/astral-sh/uv/issues/1710 (a proxy, and using invoke, etc.). That setup means that on different team member's machine's, each is likely to have a different INDEX_URL and even the CI/CD system we use would have a different one.

I've generated the uv.lock file using UV_INDEX_URL=... uv lock, and the main thing I noticed is that the index url is included in the lock file e.g.

[[package]]
name = "aws-embedded-metrics"
version = "3.2.0"
source = { registry = "http://<omitted>/pypi/simple" }
dependencies = [
    { name = "aiohttp" },
]
sdist = { url = "http://<omitted>/pypi/simple/aws-embedded-metrics/3.2.0/aws-embedded-metrics-3.2.0.tar.gz", hash = "sha256:f235f87ab25ff328f6f3afca1c6b3218e81eea6e96e6aee012d368bb813fae7b" }
wheels = [
    { url = "http://<omitted>/pypi/simple/aws-embedded-metrics/3.2.0/aws_embedded_metrics-3.2.0-py3-none-any.whl", hash = "sha256:887b76d24914efa5fc42a7b77983e77fc670633e6e1195aac7653c425fee7399" },
]

I have not tried using that file across our different machines yet, but I suspect it's going to cause problems so I wanted to confirm with you folks

• What would happen if the index in the lock file is not available, and UV_INDEX_URL?
• Is there a way to not include the full index url in the lock file?

[humanzz]

I've tested it further and confirmed that uv sync would fail on the different machines given the UV_INDEX_URL= would be pointing to a different index url.

With that that, I think this is a request to enable not writing the index url into the lock files via uv lock, and to enable uv sync - and other commands to use the lock file that does not have index urls as part of the lockfile

and to be more specific, this is about

• source.registry
• sdist.url
• wheels.url

fields

[mgab]

I discovered uv recently and I'm pretty excited about it! So even if we're discussing issues, thanks! 🫶

Now, coming to the topic, I think this becomes important given the ability to setup index-url and extra-index-url as user-level configuration, and potentially as system-level configuration (#6742).

Right now if you have a user-level config pointing at a package repository that acts as proxy to PyPI (that might add private pacakges on top) you will record that url in any lock file you generate. So if you want to work on some public project where uv is used as pacakge manager, either

1. the project -that you might not control- specifically defines index-url and extra-index-url to point at PyPI
2. you disable the user-level config
3. or you will generate lock files that people outside of your org will not be able to use

Option 2 might not seem that bad, but then that defies the purpose of being able to define user-level or system-level configs. Plus, it might not be the ideal setup ideal to take advantage of uvx and uv tool using private packages that are only available in the private repository.

[humanzz]

Just adding a couple more comments

• I did see on some other issue/pr, a mention that thanks to including those full paths, uv wouldn't be doing any further resolutions, and it just goes ahead to install things - which would save some time which sounds great
• In the case of the setup we're using, I noticed something which might be useful to call out
  • when the index, is public pypi index i.e. https://pypi.org/simple, the package artifacts have urls with a prefix looking like https://files.pythonhosted.org/packages/c1/08/1bb1f7392cd9c3a65c4571a7f7286056ec52b3cd3d432485ca2b9907e0f9/. For example, for aws-embedded-metrics, it's https://files.pythonhosted.org/packages/c1/08/1bb1f7392cd9c3a65c4571a7f7286056ec52b3cd3d432485ca2b9907e0f9/aws-embedded-metrics-3.2.0.tar.gz
  • in my case, this custom index, which uses a local proxy on dev/ci machines i.e. http://<omitted>/pypi/simple, the package srtifacts urls follow a more deterministic url pattern e.g. http://<omitted>/pypi/simple/<package name>/<package version>/. The same aws-embedded-metrics would be http://<omitted>/pypi/simple/aws-embedded-metrics/3.2.0/aws-embedded-metrics-3.2.0.tar.gz.

This determinism, can actually lead to a simpler handling of this, where I tested with

1. Generate the uv.lock with setting a UV_INDEX_URL to my custom index
2. Used it in other machines, by running another small script, prior to uv sync, that modifies uv.lock to replace the index in http://<omitted>/pypi/simple with the content of UV_INDEX_URL

All of this, makes me wonder if there's some solution to cases like mine, along the lines of

• Executing commands like uv lock, to have a flag to indicate usage of placeholder index urls, where it would have a place holder for UV_INDEX_URL or the extra indices
• Executing commands like uv sync can take a similar flag, where it'd use the values passed through UV_INDEX_URL to resolve the placeholder values in uv.lock

[smheidrich]

What follows is a +1 comment you should all ignore, but I'm writing it to include some keywords in this issue so other people who run into this can find it more easily than I just did (took me a while):

---

Having the same issue because I fetch all my PyPI packages through a local devpi^(keyword 1) installation, so all source.registry and sdist.url URLs in uv.lock say http://localhost:3141/...^(keyword 2) and, needless to say, this results in Connection refused^(keyword 3) errors when trying to install the package elsewhere, e.g. CI systems such as GitHub Actions^(keyword 4) or GitLab CI/CD^(keyword 5).

[humanzz]

The more comments this gets, the more I think there needs to be an option for the lock file to not include any index information.

This way, all those use cases that use local proxies, different on developer machines from those in CI environments, and the likelihood that those different proxies having different conventions for their asset (wheel/sdist) URLs, that a general simple solution to exclude those index/asset URLs would be very useful and enable them to adopt uv.lock

[tapetersen]

We could really also use this to be able to have lock-files without disclosing internal urls. If not in the sync file then an option to the export subcommand to export a new uv.lock but with this or other things omitted would also be a good alternative

[charliermarsh]

I'm not sure that a lockfile without URLs is a good idea. One of the goals of the lockfile is to be hermetic: you can install from just the lockfile without making any queries or relying on any external sources.

If you're going to omit source information like that, why not just export to requirements.txt?

[tapetersen]

@charliermarsh That may very well work for our requirements but it looks like the uv lockfile is richer (all the extras are declared). Otherwise it would just be a convenience to not have someone outside (that don't have access to internal dns-names) being able to use the lock-file without having to regenerate it from the requirements.txt (pardon me if I've missed a quicker way to handle that).

Basically my thought was that the hashes should be enough to guarantee that it's indeed the same packages that are being installed even if the urll has to be found again.

That said if it doesn't feel like the right way to do it and the recommended way is to manage that with requirements.txt exports we'll try that and raise a more specific issue or discussion if a showstopper or question regarding that comes up.

(feel free to edit/move/hide post if it derails original thread)

[humanzz]

@charliermarsh

At the moment, I'm actually using uv pip compile pyproject.toml... and uv pip install -r ... to simulate having a lockfile.

The main thing I don't like about it, is keeping the lockfile up to date with pyproject.toml dependencies is all manual. I didn't want to always call uv pip compile, then follow immediately by uv pip install as that would (1) be slower (2) force updating resolved dependencies that are declared via version ranges.

So, I'd say that's not on par with the experience of something like package.json and package-lock.json for node, and I was hoping that uv.lock and the commands that maintain/use it are giving a better experience.

The other thing I like about uv.lock is uv.tool.dev-dependencies. Without that, I end up declaring those dependencies as optional dependencies which is really not right, but ended up doing it this way, so I can declare all my dependencies in pyproject.toml, and then generate the current requirements.txt "lockfile".

Omitting the URLs is one potential solution - me thinking out loud.

In the case of my proxy, given that the sdist/wheel URLs have a specific pattern with the index url being a prefix, I was able to run some code to manipulate the lock file before running uv sync --all-extras --frozen to install dependencies

@task
def update_lock_index_url(ctx):
    """
    Workaround for `uv lock` writing the index url to the lock file.
    Update `source` and `url` fields with values based on `UV_INDEX_URL` environment variable.
    """
    with open("uv.lock", "r+") as file:
        contents = file.read()
        pattern = r"http://127.0.0.1:\d+/[^/]+/[^/]+/[^/]+/pypi/simple"
        replacement = os.environ.get("UV_INDEX_URL", "NO_INDEX_OVERRIDE_SET")
        new_contents = re.sub(pattern, replacement, contents)
        file.seek(0)
        file.truncate()
        file.write(new_contents)

Downside for this, is that any dev on my team will end up having modifying uv.lock. 2 thoughts I had about this

1. uv having this concept of a hook - some python code to execute to modify file content - before installing dependencies
2. Ability to specify path to a lockfile, whereby I can run the hook myself, to generate a modified version of the lockfile, make sure it's in .gitignore, or in some tmp directory, and use uv sync while pointing to my modified version

I thought the simplest of all of the above would be this whole omission of URLs from the file tbh. Moreover, generally when I think about what's uv story for supporting specifying index/extra index, and how that interacts with commands like uv sync, I think the strong assumption there is that the index URLs are not changing, which at least in my case is not true - whether across dev machines, or in the CI environment.

[mgab]

I'm not sure that a lockfile without URLs is a good idea. One of the goals of the lockfile is to be hermetic: you can install from just the lockfile without making any queries or relying on any external sources.

If you're going to omit source information like that, why not just export to requirements.txt?

As far as I can see, poetry records the file name and the hash, which should be enough to ensure that you are getting exactly the same files without relying on external sources.

On the other hand, by including the URLs you are coupling the lock file with the specific path of the package repository that you are using. And while this might remove the need of interacting with the package index, it also seems that adding a proxy package repository on the working environment should be transparent to the dependency specification of my project.

[charliermarsh]

I consider it fairly critical that the lockfile includes the URLs directly and I suspect that the lockfile standard (should it be accepted) will do the same. We do have a plan to support these use-cases though. After #7481, we'll add something like:

[[tool.uv.index]]
name = "private"
url = "https://private.org/simple"
proxy = "http://<omitted>/pypi/simple"

So you can set the index URL (which will appear in the lockfile) and then the proxy separately. You'll also be able to set the proxy as an environment variable, like UV_PRIVATE_PROXY_URL or similar.

[humanzz]

Hey @charliermarsh,

Just to make sure I understand this, and try to assess whether it'd work.

• The url in our case is not needed, though I dunno whether the intention is for it to be a valid index, or any url so maybe you can clarify that.
• I guess the url is what would be saved in index? and the proxy is kinda what replaces it when doing the actual resolving/downloads?
• for the package urls e.g. sdist, and wheels, how would that work with the proxy? In my earlier messages, I did see https://pypi.org/simple has sdist and wheels that don't have the index as part of their urls e.g. https://files.pythonhosted.org/packages/c1/08/1bb1f7392cd9c3a65c4571a7f7286056ec52b3cd3d432485ca2b9907e0f9/aws-embedded-metrics-3.2.0.tar.gz but in my case, it does have the index in the url e.g. http://<omitted>/pypi/simple/aws-embedded-metrics/3.2.0/aws-embedded-metrics-3.2.0.tar.gz

[charliermarsh]

1. Ideally it is a valid index and indicative of the actual locations of the files but that wouldn't be strictly required.
2. Correct.
3. Both the file URLs and the registry URLs would be "replaced" correctly as long as they're on the same domain. So this wouldn't work for PyPI but would work for your case.

[humanzz]

Great... then I look forwards to that :)

[mgab]

I consider it fairly critical that the lockfile includes the URLs directly and I suspect that the lockfile standard (should it be accepted) will do the same.

Sorry! Actually, reading about the proposed standard I found this comment that properly describes what was my concern, and then the answer to it. The point that I was missing is in fact addressed in the PEP draft:

• Installers MUST NOT assume the URL will always work, but installers MAY use the URL if it happens to work.

So yeah, I understood that URLs would be assumed to work, and that didn't seem a great idea. I see that's not the case, so nothing to object!

What got me confused is that apparently uv sync does assume the URLs to work unless you explicitly configure a value for either index-url or extra-index-url somewhere. I reported it in #8076 and #8074, hope it's useful. In any case, that's just about keeping improving uv and not a design disagreement.

Thanks for the answers!

[humanzz]

I saw that the revamped index support got released in 0.4.23.

Is there any separate issues to track the proxy/env variables support, or is this issue itself that?

[charliermarsh]

I think it’s ok to track it here.

[rd-danny-fleer]

May I ask what's the current status of the issue? What aspects remain unclear?

Currently I use a workaround (Run uv lock once in the CICD pipeline with the alternative index URL and use the updated uv.lock file in the whole pipeline) to make CICD work with an alternative index. However, with this approach, you must be careful not to update any dependencies.

[charliermarsh]

I don't know if anything is unclear, it's more that it hasn't been prioritized.

[dariocurr]

Any updates on this? This is critical for us to ship uv in our pipelines

[humanzz]

I know it's already February, but it would still make a great New Year's present! 😄

[jtfmumm]

I've created a draft PR for this issue. The current design adds a new command line arg (--index-proxy-url <index-name>=<proxy-url>) and env var (UV_INDEX_PROXY_URL=<index-name>=<proxy-url>).

You define a canonical URL for an index by defining the index the normal way in pyproject.toml:

 [[tool.uv.index]]
name = "private"
url = "https://private.org/simple"

When passing in a proxy URL for an index, that URL will be used for making calls to the registry. However, only the canonical URL will be written to the lockfile. If the canonical URL is also valid, then you can run commands like uv sync without a proxy URL. If it's not valid, you will need to pass in the proxy URL.

This is supported for uv run, uv add, uv sync, and uv lock.

One tricky aspect (mentioned above) is that this will only work if your index uses its URL as the base URL for the source distributions and wheels. So as Charlie indicated, this doesn't work for PyPi, for example.

I'd be curious to hear any feedback about whether this addresses your concerns and whether the API makes sense. I'd also be curious if anyone would be interested in trying out my branch with their projects.

[apollo13]

Hi @jtfmumm, thank you for your work on this! Since you asked for feedback… ;)

The current design adds a new command line arg (--index-proxy-url =) and env var (UV_INDEX_PROXY_URL==).

This works only for a single index if I understand the code correctly? I guess this will work for the majority of cases and at least --index-proxy-url could be specified multiple times. Not sure how that would work for UV_INDEX_PROXY_URL though.

One tricky aspect (https://github.com/astral-sh/uv/issues/6349#issuecomment-2393167729) is that this will only work if your index uses its URL as the base URL for the source distributions and wheels. So as Charlie https://github.com/astral-sh/uv/issues/6349#issuecomment-2393409286, this doesn't work for PyPI, for example.

That is the biggest downside for me. We usually try to mirror PyPI to save bandwidth (for PyPI). I wonder if supporting something like PDM has would make sense: https://pdm-project.org/latest/usage/lockfile/#static-urls -- PDM has an option to only store the filename in the lockfile which probably means more requests to the index at some point (?).

[jtfmumm]

Hi @apollo13, thanks for the feedback!

Currently you can specify --index-proxy-url multiple times. We're going to change the env var to be of the form UV_INDEX_<index-name>_PROXY_URL. That is how we currently handle username and password env vars for indexes.

PDM has an option to only store the filename in the lockfile

Charlie raises the concern above that URLs may be required once the lockfile standards are accepted, so I'm not sure if this option is open to us.

[apollo13]

Charlie raises the concern above that URLs may be required once the lockfile standards are accepted, so I'm not sure if this option is open to us.

Understood, I wonder if we can come up with other creative ideas. Because using a proxy for PyPI in CI is the one situation where I see massive bandwidth savings for PyPI.

[charliermarsh]

Understood, I wonder if we can come up with other creative ideas. Because using a proxy for PyPI in CI is the one situation where I see massive bandwidth savings for PyPI.

When you proxy PyPI in this way, are you also proxying the returned file URLs? Or just the Simple index API? Either way, I think this should still be fine with the proposed API, since... if you're just using the file URLs returned by PyPI, they don't need authentication; and if you're proxying those, then in theory you can use a domain that's the same as the top-level Simple API domain.

[apollo13]

The returned URLs are rewritten. Ie it is not a simple HTTP proxy but a repository manager like sonatype nexus, resulting in this download link for a package:

https://my_host/repository/pypi-proxy/packages/packaging/24.2/packaging-24.2-py3-none-any.whl

which makes the URL pretty much different from PyPI which has it at:

https://files.pythonhosted.org/packages/88/ef/eb23f262cca3c0c4eb7ab1933c3b1f03d021f2c48f54763065b6f0e321be/packaging-24.2-py3-none-any.whl

[charliermarsh]

But does https://my_host/repository/pypi-proxy/packages/packaging/24.2/packaging-24.2-py3-none-any.whl match the host of the registry itself, i.e., the thing you pass to --index?

[apollo13]

I guess so, ie the simple index URL for the package packaging would be:

https://my_host/repository/pypi-proxy/simple/packaging

(my_host being the same as above). But I still don't understand why you think this would work. If I use the proxy to create the lockfile it would write package urls ala .../packaging/24.2/packaging-24.2-py3-none-any.whl into the lockfile and I don't think replacing that to the canonical URL would work.

EDIT:// if we are going off-topic here just ping me on discord.

[charliermarsh]

Ah I see. So you want to use a proxy for your own development, but put public URLs in the lockfile itself. That seems much more challenging, since we'd need to query PyPI for every package to get those URLs.