uv icon indicating copy to clipboard operation
uv copied to clipboard

pydantic<2.0.0 installs pydantic=2.0b3 when prereleases are allowed

Open erdembanak opened this issue 1 year ago • 19 comments

Using uv 0.1.3, when trying to install pydantic<2.0 with prerelease allow, uv installs pydantic==2.0b3. According to pubgrub, 2.0b3 shouldn't be contained in the set:

"For example, "2.0.0-beta" is meant to exist previous to version "2.0.0". Yet, it is not supposed to be contained in the set described by 1.0.0 <= v < 2.0.0, and only within sets where one of the bounds contains a pre-release marker such as 2.0.0-alpha <= v < 2.0.0".

Running uv pip install --prerelease=allow "pydantic<2.0" --verbose provides the below output:

 uv::requirements::from_source source=pydantic<2.0
    0.002072s DEBUG uv_interpreter::virtual_env Found a virtualenv through VIRTUAL_ENV at: /home/erdemb/.venv
    0.002191s DEBUG uv_interpreter::interpreter Using cached markers for: /home/erdemb/.venv/bin/python
    0.002207s DEBUG uv::commands::pip_install Using Python 3.10.12 environment at /home/erdemb/.venv/bin/python
 uv_client::flat_index::from_entries 
 uv_resolver::resolver::solve 
      0.003246s   0ms DEBUG uv_resolver::resolver Solving with target Python version 3.10.12
   uv_resolver::resolver::choose_version package=root
   uv_resolver::resolver::get_dependencies package=root, version=0a0.dev0
        0.003287s   0ms DEBUG uv_resolver::resolver Adding direct dependency: pydantic<2.0
   uv_resolver::resolver::choose_version package=pydantic
     uv_resolver::resolver::package_wait package_name=pydantic
 uv_resolver::resolver::process_request request=Versions pydantic
   uv_client::registry_client::simple_api package=pydantic
     uv_client::cached_client::get_cacheable 
       uv_client::cached_client::read_and_parse_cache file=/home/erdemb/.cache/uv/simple-v1/pypi/pydantic.rkyv
 uv_resolver::resolver::process_request request=Prefetch pydantic <2.0
          0.004614s   1ms  WARN uv_client::cached_client Broken cache entry at /home/erdemb/.cache/uv/simple-v1/pypi/pydantic.rkyv, removing: failed to open file `/home/erdemb/.cache/uv/simple-v1/pypi/pydantic.rkyv`
          0.004832s   1ms DEBUG uv_client::cached_client No cache entry for: https://pypi.org/simple/pydantic/
       uv_client::cached_client::fresh_request url="https://pypi.org/simple/pydantic/"
       uv_client::cached_client::new_cache file=/home/erdemb/.cache/uv/simple-v1/pypi/pydantic.rkyv
       uv_client::registry_client::parse_simple_api package=pydantic
 uv_resolver::version_map::from_metadata 
   uv_distribution::distribution_database::get_or_build_wheel_metadata dist=pydantic==2.0b3
     uv_client::registry_client::wheel_metadata built_dist=pydantic==2.0b3
       uv_client::cached_client::get_serde 
         uv_client::cached_client::get_cacheable 
           uv_client::cached_client::read_and_parse_cache file=/home/erdemb/.cache/uv/wheels-v0/pypi/pydantic/pydantic-2.0b3-py3-none-any.msgpack
        0.967483s 964ms DEBUG uv_resolver::resolver Searching for a compatible version of pydantic (<2.0)
        0.967501s 964ms DEBUG uv_resolver::resolver Selecting: pydantic==2.0b3 (pydantic-2.0b3-py3-none-any.whl)
   uv_resolver::resolver::get_dependencies package=pydantic, version=2.0b3
     uv_resolver::resolver::distributions_wait package_id=pydantic-2.0b3
              0.967626s   0ms DEBUG uv_client::cached_client Found fresh response for: https://files.pythonhosted.org/packages/af/ce/dabc6cd90abaf903d84a329f29a5ce34643fd66ac2a78f3eb1c668bd4fb3/pydantic-2.0b3-py3-none-any.whl
        0.967715s   0ms DEBUG uv_resolver::resolver Adding transitive dependency: annotated-types>=0.4.0
        0.967749s   0ms DEBUG uv_resolver::resolver Adding transitive dependency: pydantic-core==0.39.0
        0.967804s   0ms DEBUG uv_resolver::resolver Adding transitive dependency: typing-extensions>=4.6.1
   uv_resolver::resolver::choose_version package=annotated-types
     uv_resolver::resolver::package_wait package_name=annotated-types
 uv_resolver::resolver::process_request request=Versions annotated-types
   uv_client::registry_client::simple_api package=annotated-types
     uv_client::cached_client::get_cacheable 
       uv_client::cached_client::read_and_parse_cache file=/home/erdemb/.cache/uv/simple-v1/pypi/annotated-types.rkyv
 uv_resolver::resolver::process_request request=Versions pydantic-core
   uv_client::registry_client::simple_api package=pydantic-core
     uv_client::cached_client::get_cacheable 
       uv_client::cached_client::read_and_parse_cache file=/home/erdemb/.cache/uv/simple-v1/pypi/pydantic-core.rkyv
 uv_resolver::resolver::process_request request=Versions typing-extensions
   uv_client::registry_client::simple_api package=typing-extensions
     uv_client::cached_client::get_cacheable 
       uv_client::cached_client::read_and_parse_cache file=/home/erdemb/.cache/uv/simple-v1/pypi/typing-extensions.rkyv
 uv_resolver::resolver::process_request request=Prefetch typing-extensions >=4.6.1
 uv_resolver::resolver::process_request request=Prefetch pydantic-core ==0.39.0
 uv_resolver::resolver::process_request request=Prefetch annotated-types >=0.4.0
          0.967990s   0ms  WARN uv_client::cached_client Broken cache entry at /home/erdemb/.cache/uv/simple-v1/pypi/annotated-types.rkyv, removing: failed to open file `/home/erdemb/.cache/uv/simple-v1/pypi/annotated-types.rkyv`
          0.968012s   0ms  WARN uv_client::cached_client Broken cache entry at /home/erdemb/.cache/uv/simple-v1/pypi/pydantic-core.rkyv, removing: failed to open file `/home/erdemb/.cache/uv/simple-v1/pypi/pydantic-core.rkyv`
          0.969678s   1ms  WARN uv_client::cached_client Broken cache entry at /home/erdemb/.cache/uv/simple-v1/pypi/typing-extensions.rkyv, removing: failed to open file `/home/erdemb/.cache/uv/simple-v1/pypi/typing-extensions.rkyv`
          0.969754s   1ms DEBUG uv_client::cached_client No cache entry for: https://pypi.org/simple/pydantic-core/
       uv_client::cached_client::fresh_request url="https://pypi.org/simple/pydantic-core/"
          0.969908s   2ms DEBUG uv_client::cached_client No cache entry for: https://pypi.org/simple/annotated-types/
       uv_client::cached_client::fresh_request url="https://pypi.org/simple/annotated-types/"
          0.969967s   2ms DEBUG uv_client::cached_client No cache entry for: https://pypi.org/simple/typing-extensions/
       uv_client::cached_client::fresh_request url="https://pypi.org/simple/typing-extensions/"
       uv_client::cached_client::new_cache file=/home/erdemb/.cache/uv/simple-v1/pypi/pydantic-core.rkyv
       uv_client::registry_client::parse_simple_api package=pydantic-core
       uv_client::cached_client::new_cache file=/home/erdemb/.cache/uv/simple-v1/pypi/typing-extensions.rkyv
       uv_client::registry_client::parse_simple_api package=typing-extensions
 uv_resolver::version_map::from_metadata 
   uv_distribution::distribution_database::get_or_build_wheel_metadata dist=typing-extensions==4.10.0rc1
     uv_client::registry_client::wheel_metadata built_dist=typing-extensions==4.10.0rc1
       uv_client::cached_client::get_serde 
         uv_client::cached_client::get_cacheable 
           uv_client::cached_client::read_and_parse_cache file=/home/erdemb/.cache/uv/wheels-v0/pypi/typing-extensions/typing_extensions-4.10.0rc1-py3-none-any.msgpack
              1.050399s   0ms DEBUG uv_client::cached_client Found fresh response for: https://files.pythonhosted.org/packages/d4/b2/4980568cd3814002f1c04b542e2dd52faa0cec11bd5bda99e43eefe9a808/typing_extensions-4.10.0rc1-py3-none-any.whl
       uv_client::cached_client::new_cache file=/home/erdemb/.cache/uv/simple-v1/pypi/annotated-types.rkyv
       uv_client::registry_client::parse_simple_api package=annotated-types
 uv_resolver::version_map::from_metadata 
   uv_distribution::distribution_database::get_or_build_wheel_metadata dist=annotated-types==0.6.0
     uv_client::registry_client::wheel_metadata built_dist=annotated-types==0.6.0
       uv_client::cached_client::get_serde 
         uv_client::cached_client::get_cacheable 
           uv_client::cached_client::read_and_parse_cache file=/home/erdemb/.cache/uv/wheels-v0/pypi/annotated-types/annotated_types-0.6.0-py3-none-any.msgpack
        1.184443s 216ms DEBUG uv_resolver::resolver Searching for a compatible version of annotated-types (>=0.4.0)
        1.184461s 216ms DEBUG uv_resolver::resolver Selecting: annotated-types==0.6.0 (annotated_types-0.6.0-py3-none-any.whl)
   uv_resolver::resolver::get_dependencies package=annotated-types, version=0.6.0
     uv_resolver::resolver::distributions_wait package_id=annotated-types-0.6.0
              1.184758s   0ms DEBUG uv_client::cached_client Found fresh response for: https://files.pythonhosted.org/packages/28/78/d31230046e58c207284c6b2c4e8d96e6d3cb4e52354721b944d3e1ee4aa5/annotated_types-0.6.0-py3-none-any.whl
   uv_resolver::resolver::choose_version package=pydantic-core
     uv_resolver::resolver::package_wait package_name=pydantic-core
 uv_resolver::version_map::from_metadata 
   uv_distribution::distribution_database::get_or_build_wheel_metadata dist=pydantic-core==0.39.0
     uv_client::registry_client::wheel_metadata built_dist=pydantic-core==0.39.0
       uv_client::cached_client::get_serde 
         uv_client::cached_client::get_cacheable 
           uv_client::cached_client::read_and_parse_cache file=/home/erdemb/.cache/uv/wheels-v0/pypi/pydantic-core/pydantic_core-0.39.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.msgpack
        6.767120s   5s  DEBUG uv_resolver::resolver Searching for a compatible version of pydantic-core (==0.39.0)
        6.767138s   5s  DEBUG uv_resolver::resolver Selecting: pydantic-core==0.39.0 (pydantic_core-0.39.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl)
   uv_resolver::resolver::get_dependencies package=pydantic-core, version=0.39.0
     uv_resolver::resolver::distributions_wait package_id=pydantic-core-0.39.0
              6.767260s   0ms DEBUG uv_client::cached_client Found fresh response for: https://files.pythonhosted.org/packages/10/15/82a7866bd5660dd4251c87ca23b01b8ba495ea0b7013bb8ee884dfce0311/pydantic_core-0.39.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
        6.767349s   0ms DEBUG uv_resolver::resolver Adding transitive dependency: typing-extensions*
   uv_resolver::resolver::choose_version package=typing-extensions
     uv_resolver::resolver::package_wait package_name=typing-extensions
        6.767401s   0ms DEBUG uv_resolver::resolver Searching for a compatible version of typing-extensions (>=4.6.1)
        6.767406s   0ms DEBUG uv_resolver::resolver Selecting: typing-extensions==4.10.0rc1 (typing_extensions-4.10.0rc1-py3-none-any.whl)
   uv_resolver::resolver::get_dependencies package=typing-extensions, version=4.10.0rc1
     uv_resolver::resolver::distributions_wait package_id=typing-extensions-4.10.0rc1
Resolved 4 packages in 6.76s
    6.767756s DEBUG uv_installer::plan Requirement already cached: annotated-types==0.6.0
    6.767789s DEBUG uv_installer::plan Requirement already cached: pydantic==2.0b3
    6.767807s DEBUG uv_installer::plan Requirement already cached: pydantic-core==0.39.0
    6.767828s DEBUG uv_installer::plan Requirement already cached: typing-extensions==4.10.0rc1
 uv_installer::installer::install num_wheels=4
Installed 4 packages in 14ms
 + annotated-types==0.6.0
 + pydantic==2.0b3
 + pydantic-core==0.39.0
 + typing-extensions==4.10.0rc1

Pubgrub documentation also says that this is a difficult problem; therefore the expected behaviour of the --prerelease=allow should be documented if this is the desired output.

erdembanak avatar Feb 18 '24 11:02 erdembanak

How Pip behaves with exclusive operators only:

Does not install pydantic prerelease: pip install --dry-run --pre "pydantic < 2.0" Does install pydantic prerelease: pip install --dry-run --pre "pydantic < 2.0rc1"

How Pip behaves with inclusive operators on June 18th 2023 (when there was only pre-releases and no final 2.0, once the final release happened inclusive operators install 2.0 final, and exclusive operators are the same):

Does install pydantic prerelease: pip install --dry-run --pre "pydantic <= 2.0" Does install pydantic prerelease: pip install --dry-run --pre "pydantic <= 2.0rc1"

The relevant parts of the Python spec:

  • https://packaging.python.org/en/latest/specifications/version-specifiers/#handling-of-pre-releases
  • https://packaging.python.org/en/latest/specifications/version-specifiers/#inclusive-ordered-comparison
  • https://packaging.python.org/en/latest/specifications/version-specifiers/#exclusive-ordered-comparison

Some recent discussion:

  • https://discuss.python.org/t/handling-of-pre-releases-when-backtracking
  • https://github.com/pypa/packaging/issues/776
  • https://github.com/pypa/pip/issues/12469
  • https://github.com/pypa/pip/issues/12471
  • https://github.com/pypa/pip/issues/12472

IMO the Python spec leads to ambigious edge cases, I think both uv's and Pip's behavior here can be interpreted to be part of the spec.

notatallshaw avatar Feb 18 '24 15:02 notatallshaw

According to pubgrub

@erdembanak what pubgrub doc states in that page is that the semantic meaning of "<2.00" is different than its mathematic, which makes solving difficult with solvers that cannot alter the knowledge it gathers while solving depending on context. For this reason, the resolver has to take decisions that are context independent. In the case of uv, as stated in their readme https://github.com/astral-sh/uv?tab=readme-ov-file#pre-release-handling, they choose the two rules:

  1. If the package is a direct dependency, and its version markers include a pre-release specifier (e.g., flask>=2.0.0rc1).
  2. If all published versions of a package are pre-releases.

But if you --prerelease=allow then it allows pre-releases for all dependencies. In your case, 2.0b3 is mathematically < 2.0 and you allowed pre-release for all, so it seems normal that it installs the prerelease.

mpizenberg avatar Feb 18 '24 16:02 mpizenberg

I do not think the spec is ambiguous

The exclusive ordered comparison <V MUST NOT allow a pre-release of the specified version unless the specified version is itself a pre-release.

(bolded capital letters from the original)

dimbleby avatar Feb 18 '24 16:02 dimbleby

Also in the spec: https://packaging.python.org/en/latest/specifications/version-specifiers/#handling-of-pre-releases

Dependency resolution tools SHOULD also allow users to request the following alternative behaviours:

  • accepting pre-releases for all version specifiers
  • excluding pre-releases for all version specifiers (reporting an error or warning if a pre-release is already installed locally, or if a pre-release is the only way to satisfy a particular specifier)

mpizenberg avatar Feb 18 '24 16:02 mpizenberg

I suggest that the way to read this so that the spec is consistent is: the section about handling pre-releases is discussing the behaviour for those specifiers that only implicitly disallow pre-releases.

Exclusive ordered comparisons very clearly and very explicitly always disallow pre-releases.

dimbleby avatar Feb 18 '24 16:02 dimbleby

I do not think the spec is ambiguous

The exclusive ordered comparison <V MUST NOT allow a pre-release of the specified version unless the specified version is itself a pre-release.

What's ambiguous is when the resolver specifies the prerelease flag, does that implicitly mean all versions include prerelease?

If it does the uv is following the spec and pip is not.

If it doesn't what does it even mean to include a prerelease flag?

notatallshaw avatar Feb 18 '24 17:02 notatallshaw

If it doesn't what does it even mean to include a prerelease flag?

see the previous comment.

I believe that the section about handling pre-releases is describing an option for eg the inclusive ordered comparisons

  • for those, the comparison operator in itself does not exclude pre-releases
  • but pre-releases are nevertheless in general implicitly excluded
  • an "allow-prereleases" flag is intended to overrule that general implicit exclusion
  • such a flag is not intended to overrule the basic meaning of a comparison eg the exclusive ordered comparison - regardless of any general implicit rules - always explicitly excludes prereleases

Eg ==2.0.0 does not accept 2.0b3 just because "allow-prereleases" is set: the version 2.0b3 does not satisfy the meaning of ==2.0.0. Similar reasoning holds for <2.0.0

dimbleby avatar Feb 18 '24 17:02 dimbleby

That's certainly a way to interpret the spec, but it doesn't explicitly state it, so one can also interpret how uv has implemented it.

notatallshaw avatar Feb 18 '24 17:02 notatallshaw

I certainly agree the spec could be clearer, but I think you have to work quite hard to justify ignoring

The exclusive ordered comparison <V MUST NOT allow a pre-release of the specified version unless the specified version is itself a pre-release.

dimbleby avatar Feb 18 '24 17:02 dimbleby

You only have to read the next sentence and take prerelease flag to mean that prerelease versions are included.

It takes only the work of correlating two things together.

notatallshaw avatar Feb 18 '24 17:02 notatallshaw

In any case, I do agree your interpretation is reasonable and it's what pip does. So probably uv maintainers want to follow what pip does when what pip does makes sense?

notatallshaw avatar Feb 18 '24 17:02 notatallshaw

Let's not fight over this. Let's wait for what the uv team has to say.

mpizenberg avatar Feb 18 '24 17:02 mpizenberg

Hmm... I think my honest read of the spec is that even with --prerelease=allow, the specifier <2.0.0 should not include 2.0b3, but that it should allow 1.9b3 based on "a pre-release of the specified version".

That being said, I think we're unlikely to implement it (at least not soon), since (1) it breaks fundamental properties of version solving, (2) makes the solver much harder to implement and reason about, and (3) frankly, may be even more confusing for users in some cases. In other words, I just doubt it will be high-priority enough for us to support it in the near future.

charliermarsh avatar Feb 18 '24 18:02 charliermarsh

I suppose I'd also add (in our defense) that I wouldn't be surprised if there is no Python resolver that is fully spec compliant on pre-releases. (I can't say this definitively, I haven't looked enough.) In particular, the last clause here is very hard to satisfy, since it means you have to expand the set of candidate packages as you proceed with a resolution:

Pre-releases of any kind, including developmental releases, are implicitly excluded from all version specifiers, unless they are already present on the system, explicitly requested by the user, or if the only available version that satisfies the version specifier is a pre-release.

(I'm also not sure how that clause is intended to be interpreted. For example, say you have releases for Pydantic at 1.8.0, 1.9.0a, and 2.1.0 (but no other releases). And then you have one package that requests Pydantic at <2.0.0, and another that requests Pydantic at !=1.8.0. What should happen? Both markers can be individually satisfied without a pre-release, but taken together, they can only be satisfied with a pre-release.)

charliermarsh avatar Feb 18 '24 18:02 charliermarsh

In particular, the last clause here is very hard to satisfy, since it means you have to expand the set of candidate packages as you proceed with a resolution:

Pre-releases of any kind, including developmental releases, are implicitly excluded from all version specifiers, unless they are already present on the system, explicitly requested by the user, or if the only available version that satisfies the version specifier is a pre-release.

This is discussed, at some length, in the Python thread I linked.

I believe Poetry does attempt to follow this spec, but fails under more complicated circumstances (e.g. when backtracking on transitive dependencies), and there is an open issue on Poetry side for this.

Pip does not to follow it, as linked in the various issues I posted. One of the pip maintainers characterizes the rules pip follows in the linked Python thread pretty early on. Which I believe is the approach rip is following for this.

notatallshaw avatar Feb 18 '24 18:02 notatallshaw

(Thanks, I swear I did read all of the pip issues you linked, but I didn’t make it through the entire Python thread yet.)

charliermarsh avatar Feb 18 '24 19:02 charliermarsh

I wouldn't be surprised if there is no Python resolver that is fully spec compliant

For what it's worth I don't know any other solver that considers 2.0b3 to be compatible with <2.0.0, though I'm sure they all have bugs!

... What should happen? ...

it can get even more confusing than that! What if the solver goes down a path such that dependency foo can only be satisfied with a pre-release; but - if only it had been omniscient - there was another path available such that foo did not need a pre-release? What a mess!

dimbleby avatar Feb 18 '24 19:02 dimbleby

Yeah, I'm sorry the discussion is so fragmented.

Honestly I got pretty frustrated with that topic, in particular it makes it extremely difficult to try and apply some static analysis resolver optimizations I wanted to try out, because "what is the right answer" appears to both involve a lot of contextual state about what the resolver has exhausted so far and actually be pretty ambiguous in relationship to the spec.

I am looking at packse, and may try and add some test cases that everyone can actually agree on, and then go from there. But I can't promise anything soon.

notatallshaw avatar Feb 18 '24 19:02 notatallshaw

Thanks for the comments.

Hmm... I think my honest read of the spec is that even with --prerelease=allow, the specifier <2.0.0 should not include 2.0b3, but that it should allow 1.9b3 based on "a pre-release of the specified version". ,

I agree with this understanding. Let me tell you how I run into this issue (I wasn't just messing around). This became a problem for me when I am working with an easily reproducible requirements file:

azure-cli==2.46.0 pydantic<2.0.0

If I allow prerelease, pydantic is installed as beta and if I don't, I can't install the packages due to azure-cli depending on a prerelease:

  × No solution found when resolving dependencies:
  ╰─▶ Because there is no version of azure-mgmt-sql==4.0.0b8 and azure-cli==2.46.0 depends on azure-mgmt-sql==4.0.0b8, we can conclude that
      azure-cli==2.46.0 cannot be used.
      And because you require azure-cli==2.46.0, we can conclude that the requirements are unsatisfiable.

      hint: azure-mgmt-sql was requested with a pre-release marker (e.g., azure-mgmt-sql==4.0.0b8), but pre-releases weren't enabled (try:
      `--prerelease=allow`)

Even without --pre marker, pip installs azure-cli; so this is not a problem in pip. I am currently using overrides file (giving pydantic==1.x.x in there) to overcome this problem (the pydantic constraint is not coming from my requirement file); but it is just a workaround and makes migrating to uv from pip a bit harder than it should be.

erdembanak avatar Feb 18 '24 19:02 erdembanak

Hmmm, the problem of "I want prereleases from one part of my requirements but not the other part of my requirements" is not supported by pip either, it just happens to be that the behavior pip has chose works out for you.

Unless I'm missing something fundamental about the soundness of the behavior I'm sure one could come up with another example where it would not work for pip but it would work for uv. But pip is so pervasive that everyone checks their use case works for it and if not changes their use case.

If you're interested in a workaround you can split this into multiple commands with constraints, I actually do this with pip constantly to keep my environment in sync, so this workflow will work with both uv and pip:

$ echo "azure-cli==2.46.0" | uv pip compile --prerelease=allow  - > requirements.txt
$ echo "pydantic<2.0.0" | uv pip compile --constraint requirements.txt - >> requirements.txt
$ uv pip install --requirement requirements.txt 

notatallshaw avatar Feb 19 '24 02:02 notatallshaw

If anyone interested, writing some additional test scenarios for this could be really helpful.

zanieb avatar Feb 19 '24 03:02 zanieb

just happens to be that the behavior pip has chose works out for you

I don't think that's right. The requirement azure-mgmt-sql==4.0.0b8 should find the prerelease regardless of "allow prereleases", per

... explicitly requested by the user, or if the only available version that satisfies the version specifier is a pre-release.

dimbleby avatar Feb 19 '24 08:02 dimbleby

just happens to be that the behavior pip has chose works out for you

I don't think that's right. The requirement azure-mgmt-sql==4.0.0b8 should find the prerelease regardless of "allow prereleases", per

... explicitly requested by the user, or if the only available version that satisfies the version specifier is a pre-release.

My comment about the behavior was about the choice prereleases with inclusive/exclusive operators when the prereleases flag is passed, the part of the spec you quote is highly problematic, Poetry is the only tool I'm aware of that tries to faithfully follow it but it fails to when backtracking, and I'm not even sure it's logically sound to follow when backtracking (at least using any of the satisfiability algorithms any installer currently uses).

Pip certainly doesn't: https://github.com/pypa/pip/issues/12469. And rip copies pip's behavior, at least for now: https://github.com/prefix-dev/rip/issues/118. And I think uv copies this behavior by default also? I've not looked at the logic or tested but it seems to be able to install opentelemetry-exporter-prometheus without changing the preinstall flag.

I know I posted a lot of links earlier and it was a bit much to expect everyone to read them, but this is how pip's current behavior is best characterized in how it "follows" that part of the spec: https://discuss.python.org/t/handling-of-pre-releases-when-backtracking/40505/4

notatallshaw avatar Feb 19 '24 12:02 notatallshaw

Btw, rather than discussing the ambigious, and/or poorly implemented by everyone, spec, maybe it would make more sense to just report the usability issue that pip can install this requirement, even without the prerelease flag, and uv cannot:

$ pip install --dry-run "azure-cli==2.46.0" "pydantic<2.0"
...
Would install Deprecated-1.2.14 PyGithub-1.59.1 PyJWT-2.8.0 PyMySQL-1.0.3 PyNaCl-1.5.0 PySocks-1.7.1 PyYAML-6.0.1 Pygments-2.17.2 adal-1.2.7 antlr4-python3-runtime-4.9.3 applicationinsights-0.11.10 argcomplete-2.1.2 azure-appconfiguration-1.1.1 azure-batch-13.0.0 azure-cli-2.46.0 azure-cli-core-2.46.0 azure-cli-telemetry-1.0.8 azure-common-1.1.28 azure-core-1.30.0 azure-cosmos-3.2.0 azure-data-tables-12.4.0 azure-datalake-store-0.0.53 azure-graphrbac-0.60.0 azure-keyvault-1.1.0 azure-keyvault-administration-4.0.0b3 azure-keyvault-keys-4.8.0b2 azure-loganalytics-0.1.1 azure-mgmt-advisor-9.0.0 azure-mgmt-apimanagement-3.0.0 azure-mgmt-appconfiguration-2.2.0 azure-mgmt-applicationinsights-1.0.0 azure-mgmt-authorization-3.0.0 azure-mgmt-batch-17.0.0 azure-mgmt-batchai-7.0.0b1 azure-mgmt-billing-6.0.0 azure-mgmt-botservice-2.0.0 azure-mgmt-cdn-12.0.0 azure-mgmt-cognitiveservices-13.3.0 azure-mgmt-compute-29.1.0 azure-mgmt-consumption-2.0.0 azure-mgmt-containerinstance-10.1.0 azure-mgmt-containerregistry-10.1.0 azure-mgmt-containerservice-21.2.0 azure-mgmt-core-1.4.0 azure-mgmt-cosmosdb-9.0.0 azure-mgmt-databoxedge-1.0.0 azure-mgmt-datalake-analytics-0.2.1 azure-mgmt-datalake-nspkg-3.0.1 azure-mgmt-datalake-store-0.5.0 azure-mgmt-datamigration-10.0.0 azure-mgmt-devtestlabs-4.0.0 azure-mgmt-dns-8.0.0 azure-mgmt-eventgrid-10.2.0b2 azure-mgmt-eventhub-10.1.0 azure-mgmt-extendedlocation-1.0.0b2 azure-mgmt-hdinsight-9.0.0 azure-mgmt-imagebuilder-1.1.0 azure-mgmt-iotcentral-10.0.0b2 azure-mgmt-iothub-2.3.0 azure-mgmt-iothubprovisioningservices-1.1.0 azure-mgmt-keyvault-10.1.0 azure-mgmt-kusto-0.3.0 azure-mgmt-loganalytics-13.0.0b4 azure-mgmt-managedservices-1.0.0 azure-mgmt-managementgroups-1.0.0 azure-mgmt-maps-2.0.0 azure-mgmt-marketplaceordering-1.1.0 azure-mgmt-media-9.0.0 azure-mgmt-monitor-5.0.1 azure-mgmt-msi-7.0.0 azure-mgmt-netapp-9.0.1 azure-mgmt-network-21.0.1 azure-mgmt-nspkg-3.0.2 azure-mgmt-policyinsights-1.1.0b4 azure-mgmt-privatedns-1.0.0 azure-mgmt-rdbms-10.2.0b14 azure-mgmt-recoveryservices-2.2.0 azure-mgmt-recoveryservicesbackup-5.1.0 azure-mgmt-redhatopenshift-1.2.0 azure-mgmt-redis-14.1.0 azure-mgmt-relay-0.1.0 azure-mgmt-resource-21.1.0b1 azure-mgmt-search-8.0.0 azure-mgmt-security-3.0.0 azure-mgmt-servicebus-8.2.0 azure-mgmt-servicefabric-1.0.0 azure-mgmt-servicefabricmanagedclusters-1.0.0 azure-mgmt-servicelinker-1.2.0b1 azure-mgmt-signalr-1.1.0 azure-mgmt-sql-4.0.0b8 azure-mgmt-sqlvirtualmachine-1.0.0b5 azure-mgmt-storage-21.0.0 azure-mgmt-synapse-2.1.0b5 azure-mgmt-trafficmanager-1.0.0 azure-mgmt-web-7.0.0 azure-multiapi-storage-1.0.0 azure-nspkg-3.0.2 azure-storage-common-1.4.2 azure-synapse-accesscontrol-0.5.0 azure-synapse-artifacts-0.15.0 azure-synapse-managedprivateendpoints-0.4.0 azure-synapse-spark-0.2.0 bcrypt-4.1.2 certifi-2024.2.2 cffi-1.16.0 chardet-3.0.4 charset-normalizer-3.3.2 colorama-0.4.6 cryptography-40.0.2 distro-1.9.0 fabric-2.7.1 humanfriendly-10.0 idna-3.6 invoke-1.7.3 isodate-0.6.1 javaproperties-0.5.2 jmespath-1.0.1 jsondiff-2.0.0 knack-0.10.1 msal-1.20.0 msal-extensions-1.0.0 msrest-0.7.1 msrestazure-0.6.4 oauthlib-3.2.2 packaging-23.2 paramiko-3.4.0 pathlib2-2.3.7.post1 pkginfo-1.9.6 portalocker-2.8.2 psutil-5.9.8 pyOpenSSL-23.2.0 pycparser-2.21 pydantic-1.10.14 python-dateutil-2.8.2 requests-2.31.0 requests-oauthlib-1.3.1 scp-0.13.6 semver-2.13.0 six-1.16.0 sshtunnel-0.1.5 tabulate-0.9.0 typing_extensions-4.9.0 urllib3-2.2.1 websocket-client-1.3.3 wrapt-1.16.0 xmltodict-0.13.0
$ printf "azure-cli==2.46.0\npydantic<2.0" | uv pip compile -
  × No solution found when resolving dependencies:
  ╰─▶ Because there is no version of azure-mgmt-sql==4.0.0b8 and azure-cli==2.46.0 depends on azure-mgmt-sql==4.0.0b8, we can conclude that
      azure-cli==2.46.0 cannot be used.
      And because you require azure-cli==2.46.0, we can conclude that the requirements are unsatisfiable.

      hint: azure-mgmt-sql was requested with a pre-release marker (e.g., azure-mgmt-sql==4.0.0b8), but pre-releases weren't enabled (try:
      `--prerelease=allow`)

notatallshaw avatar Feb 19 '24 13:02 notatallshaw

We can just say uv pip install azure-cli==2.46.0 is not working while pip install azure-cli==2.46.0 is working.

erdembanak-invent avatar Feb 19 '24 13:02 erdembanak-invent

So far as I know everyone tries to follow the spec, of course some are more successful than others, and in different ways.

All of the installers that I tried - pip / poetry / pdm - successfully install azure-cli==2.46.0 without extra flags, and all of them agree that 2.0b3 does not satisfy <2.0.0.

100% agree that as a user experience this issue is best reported as uv pip install azure-cli==2.46.0 does not succeed. Referring to the spec is supposed to be a useful way of investigating where this is going wrong.

dimbleby avatar Feb 19 '24 14:02 dimbleby

So, that uv pip install azure-cli==2.46.0 does not succeed is known in our design and documented. The requirement we have is that we need to know which packages are "allowed" to have pre-releases in advance. Without this, correctness becomes extremely difficult.

In the case above, we need to know that you're allowing pre-releases for azure-mgmt-sql. You can do this with:

uv pip install azure-cli==2.46.0 azure-mgmt-sql==4.0.0b8

In uv, the use of the pre-release marker in the direct dependency tells uv to allow pre-releases for that dependency. You can see this in the hint provided above upon failure.

Unfortunately, tt seems like Azure publishes, like, everything as a pre-release?

charliermarsh avatar Feb 19 '24 14:02 charliermarsh

Just before I have seen your reply, I have created a separate issue for this; sorry @charliermarsh . Yes, this is the expected behavior from the documentation, just wanted to open a separate issue to make sure the discussion doesn't get lost in here; but you already replied. Thanks.

erdembanak avatar Feb 19 '24 14:02 erdembanak

No prob. It's fair game to petition that we change the behavior, but those are the rules we settled on for now as-implemented, to make the problem tractable.

charliermarsh avatar Feb 19 '24 14:02 charliermarsh

It looks like to make sure I am not installing something as prerelease when there is a release, I need to do something like:

uv pip install azure-cli==2.46.0 azure-mgmt-sql==4.0.0b8 azure-keyvault-keys==4.8.0b2 azure-mgmt-synapse==2.1.0b5 azure-mgmt-servicelinker==1.2.0b1 azure-mgmt-batchai==7.0.0b1 azure-mgmt-extendedlocation==1.0.0b2 azure-mgmt-loganalytics==13.0.0b4 azure-mgmt-eventgrid==10.2.0b2 azure-mgmt-sqlvirtualmachine==1.0.0b5 azure-keyvault-administration==4.0.0b3 azure-mgmt-resource==21.1.0b1 azure-mgmt-rdbms==10.2.0b6 azure-mgmt-policyinsights==1.1.0b2 azure-mgmt-iotcentral==10.0.0b1

(some of them were >= in the requirements but I converted to ==)

erdembanak avatar Feb 19 '24 15:02 erdembanak