poetry icon indicating copy to clipboard operation
poetry copied to clipboard

Published 20 hours ago • verified publisher icon python-poetry

Multiple constraints on same package cause O(exp(N)) checks

Open buriy opened this issue 2 years ago • 16 comments 

poetry init -n
poetry add -vvv numpy ipython jupyter opencv-python notebook pandas django~3.2

Runs very long dependency resolution.

Problem is that it does a lot of override attempts (why?). 0: Retrying dependency resolution with the following overrides ({Package('pandas', '1.4.0'): {'numpy': <Dependency numpy (>=1.21.0)>}, Package('opencv-python', '4.5.5.62'): {'numpy': <Dependency numpy (>=1.17.3)>}, Package('black', '21.12b0'): {'typing-extensions': <Dependency typing-extensions (!=3.10.0.1)>}}).

including weird ones like 0: Retrying dependency resolution with the following overrides ({Package('pandas', '1.4.0'): {'numpy': <Dependency numpy (>=1.18.5)>}, Package('opencv-python', '4.5.5.62'): {'numpy': <Dependency numpy (>=1.21.2)>}}).

52 override checks for this case, so, if you now have several additional slow loading packages, e.g. each override checking would take 20 seconds, you'll get 20 * 52 = 520 seconds to install.

0: Complete version solving took 13.010 seconds with 52 overrides 0: Resolved with overrides: ({Package('pandas', '1.4.0'): {'numpy': <Dependency numpy (>=1.18.5)>}}), ({Package('pandas', '1.4.0'): {'numpy': <Dependency numpy (>=1.18.5)>}, Package('opencv-python', '4.5.5.62'): {'numpy': <Dependency numpy (>=1.21.2)>}}), ({Package('pandas', '1.4.0'): {'numpy': <Dependency numpy (>=1.18.5)>}, Package('opencv-python', '4.5.5.62'): {'numpy': <Dependency numpy (>=1.21.2)>}, Package('black', '21.12b0'): {'typing-extensions': <Dependency typing-extensions (>=3.10.0.0)>}}), ({Package('pandas', '1.4.0'): {'numpy': <Dependency numpy (>=1.18.5)>}, Package('opencv-python', '4.5.5.62'): {'numpy': <Dependency numpy (>=1.21.2)>}, Package('black', '21.12b0'): {'typing-extensions': <Dependency typing-extensions (!=3.10.0.1)>}}), ({Package('pandas', '1.4.0'): {'numpy': <Dependency numpy (>=1.18.5)>}, Package('opencv-python', '4.5.5.62'): {'numpy': <Dependency numpy (>=1.19.3)>}}), ({Package('pandas', '1.4.0'): {'numpy': <Dependency numpy (>=1.18.5)>}, Package('opencv-python', '4.5.5.62'): {'numpy': <Dependency numpy (>=1.19.3)>}, Package('black', '21.12b0'): {'typing-extensions': <Dependency typing-extensions (>=3.10.0.0)>}}), ({Package('pandas', '1.4.0'): {'numpy': <Dependency numpy (>=1.18.5)>}, Package('opencv-python', '4.5.5.62'): {'numpy': <Dependency numpy (>=1.19.3)>}, Package('black', '21.12b0'): {'typing-extensions': <Dependency typing-extensions (!=3.10.0.1)>}}), ({Package('pandas', '1.4.0'): {'numpy': <Dependency numpy (>=1.18.5)>}, Package('opencv-python', '4.5.5.62'): {'numpy': <Dependency numpy (>=1.14.5)>}}), ({Package('pandas', '1.4.0'): {'numpy': <Dependency numpy (>=1.18.5)>}, Package('opencv-python', '4.5.5.62'): {'numpy': <Dependency numpy (>=1.14.5)>}, Package('black', '21.12b0'): {'typing-extensions': <Dependency typing-extensions (>=3.10.0.0)>}}), ({Package('pandas', '1.4.0'): {'numpy': <Dependency numpy (>=1.18.5)>}, Package('opencv-python', '4.5.5.62'): {'numpy': <Dependency numpy (>=1.14.5)>}, Package('black', '21.12b0'): {'typing-extensions': <Dependency typing-extensions (!=3.10.0.1)>}}), ({Package('pandas', '1.4.0'): {'numpy': <Dependency numpy (>=1.18.5)>}, Package('opencv-python', '4.5.5.62'): {'numpy': <Dependency numpy (>=1.17.3)>}}), ({Package('pandas', '1.4.0'): {'numpy': <Dependency numpy (>=1.18.5)>}, Package('opencv-python', '4.5.5.62'): {'numpy': <Dependency numpy (>=1.17.3)>}, Package('black', '21.12b0'): {'typing-extensions': <Dependency typing-extensions (>=3.10.0.0)>}}), ({Package('pandas', '1.4.0'): {'numpy': <Dependency numpy (>=1.18.5)>}, Package('opencv-python', '4.5.5.62'): {'numpy': <Dependency numpy (>=1.17.3)>}, Package('black', '21.12b0'): {'typing-extensions': <Dependency typing-extensions (!=3.10.0.1)>}}), ({Package('pandas', '1.4.0'): {'numpy': <Dependency numpy (>=1.19.2)>}}), ({Package('pandas', '1.4.0'): {'numpy': <Dependency numpy (>=1.19.2)>}, Package('opencv-python', '4.5.5.62'): {'numpy': <Dependency numpy (>=1.21.2)>}}), ({Package('pandas', '1.4.0'): {'numpy': <Dependency numpy (>=1.19.2)>}, Package('opencv-python', '4.5.5.62'): {'numpy': <Dependency numpy (>=1.21.2)>}, Package('black', '21.12b0'): {'typing-extensions': <Dependency typing-extensions (>=3.10.0.0)>}}), ({Package('pandas', '1.4.0'): {'numpy': <Dependency numpy (>=1.19.2)>}, Package('opencv-python', '4.5.5.62'): {'numpy': <Dependency numpy (>=1.21.2)>}, Package('black', '21.12b0'): {'typing-extensions': <Dependency typing-extensions (!=3.10.0.1)>}}), ({Package('pandas', '1.4.0'): {'numpy': <Dependency numpy (>=1.19.2)>}, Package('opencv-python', '4.5.5.62'): {'numpy': <Dependency numpy (>=1.19.3)>}}), ({Package('pandas', '1.4.0'): {'numpy': <Dependency numpy (>=1.19.2)>}, Package('opencv-python', '4.5.5.62'): {'numpy': <Dependency numpy (>=1.19.3)>}, Package('black', '21.12b0'): {'typing-extensions': <Dependency typing-extensions (>=3.10.0.0)>}}), ({Package('pandas', '1.4.0'): {'numpy': <Dependency numpy (>=1.19.2)>}, Package('opencv-python', '4.5.5.62'): {'numpy': <Dependency numpy (>=1.19.3)>}, Package('black', '21.12b0'): {'typing-extensions': <Dependency typing-extensions (!=3.10.0.1)>}}), ({Package('pandas', '1.4.0'): {'numpy': <Dependency numpy (>=1.19.2)>}, Package('opencv-python', '4.5.5.62'): {'numpy': <Dependency numpy (>=1.14.5)>}}), ({Package('pandas', '1.4.0'): {'numpy': <Dependency numpy (>=1.19.2)>}, Package('opencv-python', '4.5.5.62'): {'numpy': <Dependency numpy (>=1.14.5)>}, Package('black', '21.12b0'): {'typing-extensions': <Dependency typing-extensions (>=3.10.0.0)>}}), ({Package('pandas', '1.4.0'): {'numpy': <Dependency numpy (>=1.19.2)>}, Package('opencv-python', '4.5.5.62'): {'numpy': <Dependency numpy (>=1.14.5)>}, Package('black', '21.12b0'): {'typing-extensions': <Dependency typing-extensions (!=3.10.0.1)>}}), ({Package('pandas', '1.4.0'): {'numpy': <Dependency numpy (>=1.19.2)>}, Package('opencv-python', '4.5.5.62'): {'numpy': <Dependency numpy (>=1.17.3)>}}), ({Package('pandas', '1.4.0'): {'numpy': <Dependency numpy (>=1.19.2)>}, Package('opencv-python', '4.5.5.62'): {'numpy': <Dependency numpy (>=1.17.3)>}, Package('black', '21.12b0'): {'typing-extensions': <Dependency typing-extensions (>=3.10.0.0)>}}), ({Package('pandas', '1.4.0'): {'numpy': <Dependency numpy (>=1.19.2)>}, Package('opencv-python', '4.5.5.62'): {'numpy': <Dependency numpy (>=1.17.3)>}, Package('black', '21.12b0'): {'typing-extensions': <Dependency typing-extensions (!=3.10.0.1)>}}), ({Package('pandas', '1.4.0'): {'numpy': <Dependency numpy (>=1.20.0)>}}), ({Package('pandas', '1.4.0'): {'numpy': <Dependency numpy (>=1.20.0)>}, Package('opencv-python', '4.5.5.62'): {'numpy': <Dependency numpy (>=1.21.2)>}}), ({Package('pandas', '1.4.0'): {'numpy': <Dependency numpy (>=1.20.0)>}, Package('opencv-python', '4.5.5.62'): {'numpy': <Dependency numpy (>=1.21.2)>}, Package('black', '21.12b0'): {'typing-extensions': <Dependency typing-extensions (>=3.10.0.0)>}}), ({Package('pandas', '1.4.0'): {'numpy': <Dependency numpy (>=1.20.0)>}, Package('opencv-python', '4.5.5.62'): {'numpy': <Dependency numpy (>=1.21.2)>}, Package('black', '21.12b0'): {'typing-extensions': <Dependency typing-extensions (!=3.10.0.1)>}}), ({Package('pandas', '1.4.0'): {'numpy': <Dependency numpy (>=1.20.0)>}, Package('opencv-python', '4.5.5.62'): {'numpy': <Dependency numpy (>=1.19.3)>}}), ({Package('pandas', '1.4.0'): {'numpy': <Dependency numpy (>=1.20.0)>}, Package('opencv-python', '4.5.5.62'): {'numpy': <Dependency numpy (>=1.19.3)>}, Package('black', '21.12b0'): {'typing-extensions': <Dependency typing-extensions (>=3.10.0.0)>}}), ({Package('pandas', '1.4.0'): {'numpy': <Dependency numpy (>=1.20.0)>}, Package('opencv-python', '4.5.5.62'): {'numpy': <Dependency numpy (>=1.19.3)>}, Package('black', '21.12b0'): {'typing-extensions': <Dependency typing-extensions (!=3.10.0.1)>}}), ({Package('pandas', '1.4.0'): {'numpy': <Dependency numpy (>=1.20.0)>}, Package('opencv-python', '4.5.5.62'): {'numpy': <Dependency numpy (>=1.14.5)>}}), ({Package('pandas', '1.4.0'): {'numpy': <Dependency numpy (>=1.20.0)>}, Package('opencv-python', '4.5.5.62'): {'numpy': <Dependency numpy (>=1.14.5)>}, Package('black', '21.12b0'): {'typing-extensions': <Dependency typing-extensions (>=3.10.0.0)>}}), ({Package('pandas', '1.4.0'): {'numpy': <Dependency numpy (>=1.20.0)>}, Package('opencv-python', '4.5.5.62'): {'numpy': <Dependency numpy (>=1.14.5)>}, Package('black', '21.12b0'): {'typing-extensions': <Dependency typing-extensions (!=3.10.0.1)>}}), ({Package('pandas', '1.4.0'): {'numpy': <Dependency numpy (>=1.20.0)>}, Package('opencv-python', '4.5.5.62'): {'numpy': <Dependency numpy (>=1.17.3)>}}), ({Package('pandas', '1.4.0'): {'numpy': <Dependency numpy (>=1.20.0)>}, Package('opencv-python', '4.5.5.62'): {'numpy': <Dependency numpy (>=1.17.3)>}, Package('black', '21.12b0'): {'typing-extensions': <Dependency typing-extensions (>=3.10.0.0)>}}), ({Package('pandas', '1.4.0'): {'numpy': <Dependency numpy (>=1.20.0)>}, Package('opencv-python', '4.5.5.62'): {'numpy': <Dependency numpy (>=1.17.3)>}, Package('black', '21.12b0'): {'typing-extensions': <Dependency typing-extensions (!=3.10.0.1)>}}), ({Package('pandas', '1.4.0'): {'numpy': <Dependency numpy (>=1.21.0)>}}), ({Package('pandas', '1.4.0'): {'numpy': <Dependency numpy (>=1.21.0)>}, Package('opencv-python', '4.5.5.62'): {'numpy': <Dependency numpy (>=1.21.2)>}}), ({Package('pandas', '1.4.0'): {'numpy': <Dependency numpy (>=1.21.0)>}, Package('opencv-python', '4.5.5.62'): {'numpy': <Dependency numpy (>=1.21.2)>}, Package('black', '21.12b0'): {'typing-extensions': <Dependency typing-extensions (>=3.10.0.0)>}}), ({Package('pandas', '1.4.0'): {'numpy': <Dependency numpy (>=1.21.0)>}, Package('opencv-python', '4.5.5.62'): {'numpy': <Dependency numpy (>=1.21.2)>}, Package('black', '21.12b0'): {'typing-extensions': <Dependency typing-extensions (!=3.10.0.1)>}}), ({Package('pandas', '1.4.0'): {'numpy': <Dependency numpy (>=1.21.0)>}, Package('opencv-python', '4.5.5.62'): {'numpy': <Dependency numpy (>=1.19.3)>}}), ({Package('pandas', '1.4.0'): {'numpy': <Dependency numpy (>=1.21.0)>}, Package('opencv-python', '4.5.5.62'): {'numpy': <Dependency numpy (>=1.19.3)>}, Package('black', '21.12b0'): {'typing-extensions': <Dependency typing-extensions (>=3.10.0.0)>}}), ({Package('pandas', '1.4.0'): {'numpy': <Dependency numpy (>=1.21.0)>}, Package('opencv-python', '4.5.5.62'): {'numpy': <Dependency numpy (>=1.19.3)>}, Package('black', '21.12b0'): {'typing-extensions': <Dependency typing-extensions (!=3.10.0.1)>}}), ({Package('pandas', '1.4.0'): {'numpy': <Dependency numpy (>=1.21.0)>}, Package('opencv-python', '4.5.5.62'): {'numpy': <Dependency numpy (>=1.14.5)>}}), ({Package('pandas', '1.4.0'): {'numpy': <Dependency numpy (>=1.21.0)>}, Package('opencv-python', '4.5.5.62'): {'numpy': <Dependency numpy (>=1.14.5)>}, Package('black', '21.12b0'): {'typing-extensions': <Dependency typing-extensions (>=3.10.0.0)>}}), ({Package('pandas', '1.4.0'): {'numpy': <Dependency numpy (>=1.21.0)>}, Package('opencv-python', '4.5.5.62'): {'numpy': <Dependency numpy (>=1.14.5)>}, Package('black', '21.12b0'): {'typing-extensions': <Dependency typing-extensions (!=3.10.0.1)>}}), ({Package('pandas', '1.4.0'): {'numpy': <Dependency numpy (>=1.21.0)>}, Package('opencv-python', '4.5.5.62'): {'numpy': <Dependency numpy (>=1.17.3)>}}), ({Package('pandas', '1.4.0'): {'numpy': <Dependency numpy (>=1.21.0)>}, Package('opencv-python', '4.5.5.62'): {'numpy': <Dependency numpy (>=1.17.3)>}, Package('black', '21.12b0'): {'typing-extensions': <Dependency typing-extensions (>=3.10.0.0)>}}), ({Package('pandas', '1.4.0'): {'numpy': <Dependency numpy (>=1.21.0)>}, Package('opencv-python', '4.5.5.62'): {'numpy': <Dependency numpy (>=1.17.3)>}, Package('black', '21.12b0'): {'typing-extensions': <Dependency typing-extensions (!=3.10.0.1)>}})

In more complicated cases it will be:

0: Duplicate dependencies for numpy 0: Different requirements found for numpy (>=1.18.5) with markers platform_machine != "aarch64" and platform_machine != "arm64" and python_version < "3.10", numpy (>=1.19.2) with markers platform_machine == "aarch64" and python_version < "3.10" and numpy (>=1.20.0) with markers platform_machine == "arm64" and python_version < "3.10". and 0: Complete version solving took 115.009 seconds with 16 overrides 0: Resolved with overrides: ({Package('opencv-python', '4.5.5.62'): {'numpy': <Dependency numpy (>=1.21.2)>}}), ({Package('opencv-python', '4.5.5.62'): {'numpy': <Dependency numpy (>=1.21.2)>}, Package('pandas', '1.4.0'): {'numpy': <Dependency numpy (>=1.18.5)>}}), ({Package('opencv-python', '4.5.5.62'): {'numpy': <Dependency numpy (>=1.21.2)>}, Package('pandas', '1.4.0'): {'numpy': <Dependency numpy (>=1.19.2)>}}), ({Package('opencv-python', '4.5.5.62'): {'numpy': <Dependency numpy (>=1.21.2)>}, Package('pandas', '1.4.0'): {'numpy': <Dependency numpy (>=1.20.0)>}}), ({Package('opencv-python', '4.5.5.62'): {'numpy': <Dependency numpy (>=1.19.3)>}}), ({Package('opencv-python', '4.5.5.62'): {'numpy': <Dependency numpy (>=1.19.3)>}, Package('pandas', '1.4.0'): {'numpy': <Dependency numpy (>=1.18.5)>}}), ({Package('opencv-python', '4.5.5.62'): {'numpy': <Dependency numpy (>=1.19.3)>}, Package('pandas', '1.4.0'): {'numpy': <Dependency numpy (>=1.19.2)>}}), ({Package('opencv-python', '4.5.5.62'): {'numpy': <Dependency numpy (>=1.19.3)>}, Package('pandas', '1.4.0'): {'numpy': <Dependency numpy (>=1.20.0)>}}), ({Package('opencv-python', '4.5.5.62'): {'numpy': <Dependency numpy (>=1.14.5)>}}), ({Package('opencv-python', '4.5.5.62'): {'numpy': <Dependency numpy (>=1.14.5)>}, Package('pandas', '1.4.0'): {'numpy': <Dependency numpy (>=1.18.5)>}}), ({Package('opencv-python', '4.5.5.62'): {'numpy': <Dependency numpy (>=1.14.5)>}, Package('pandas', '1.4.0'): {'numpy': <Dependency numpy (>=1.19.2)>}}), ({Package('opencv-python', '4.5.5.62'): {'numpy': <Dependency numpy (>=1.14.5)>}, Package('pandas', '1.4.0'): {'numpy': <Dependency numpy (>=1.20.0)>}}), ({Package('opencv-python', '4.5.5.62'): {'numpy': <Dependency numpy (>=1.17.3)>}}), ({Package('opencv-python', '4.5.5.62'): {'numpy': <Dependency numpy (>=1.17.3)>}, Package('pandas', '1.4.0'): {'numpy': <Dependency numpy (>=1.18.5)>}}), ({Package('opencv-python', '4.5.5.62'): {'numpy': <Dependency numpy (>=1.17.3)>}, Package('pandas', '1.4.0'): {'numpy': <Dependency numpy (>=1.19.2)>}}), ({Package('opencv-python', '4.5.5.62'): {'numpy': <Dependency numpy (>=1.17.3)>}, Package('pandas', '1.4.0'): {'numpy': <Dependency numpy (>=1.20.0)>}})

buriy avatar Jan 28 '22 06:01 buriy

Just an explanation what's happing:

The overrides originate from multiple constraint dependencies. In your example, there are three packages with multiple constraint dependencies:

pandas with 4 constraints for numpy

"numpy (>=1.18.5) ; platform_machine != \"aarch64\" and platform_machine != \"arm64\" and python_version < \"3.10\""
"numpy (>=1.19.2) ; platform_machine == \"aarch64\" and python_version < \"3.10\""
"numpy (>=1.20.0) ; platform_machine == \"arm64\" and python_version < \"3.10\""
"numpy (>=1.21.0) ; python_version >= \"3.10\""

opencv-python with 7 constraints for numpy

"numpy (>=1.13.3) ; python_version < \"3.7\""
"numpy (>=1.21.2) ; python_version >= \"3.10\""
"numpy (>=1.21.2) ; python_version >= \"3.6\" and platform_system == \"Darwin\" and platform_machine == \"arm64\""
"numpy (>=1.19.3) ; python_version >= \"3.6\" and platform_system == \"Linux\" and platform_machine == \"aarch64\""
"numpy (>=1.14.5) ; python_version >= \"3.7\""
"numpy (>=1.17.3) ; python_version >= \"3.8\""
"numpy (>=1.19.3) ; python_version >= \"3.9\""

These 7 constraints result in 4 actual constraints because the first one can be ignored if the python constraint of the project is >= 3.7, the second and the third as well as the fourth and the seventh can be merged by combining the markers with an or.

black with 2 constraints for typing-extensions

"typing-extensions (>=3.10.0.0)"
"typing-extensions (!=3.10.0.1) ; python_version >= \"3.10\""

Thus, the complete number of combinations is 4 * 4 * 2 == 32. Because of the way poetry counts overrides this results in 4 + 4 * 4 + 4 * 4 * 2 == 52.

radoering avatar Jan 28 '22 18:01 radoering

  1. you have only one python version installed so for example you shouldn't ever consider "python_version >= 3.10" alternative when you have python 3.8 installed. Next, after you checked for the current python version and arch (dropping unrealistic alternatives), constraints like "numpy >= 1.21.0" + "numpy >= 1.19.2" then can be simplified into just "numpy >= 1.19.2" so no "overrides" should happen.
  2. Even if you can't simplify the constraints, you shouldn't run a full check for every possible alternative if you've found a good one already. Alternatives need to be checked only when you have a conflict, so usually that's the right context for "overrides" word.
  3. And what if I added an additional constraint "numpy >= 1.22.1" manually, no other long checks should be performed at all if numpy >= 1.22.1 is ok to set up!... Unfortunately, it doesn't work that way. Guess how much time poetry add -vvv numpy~1.22 ipython jupyter opencv-python notebook pandas~1.4 typing-extensions=">=3.10.0.2 takes now (compared to example above). Same 12 seconds doing nothing useful! It can't figure out even a "numpy >= 1.22" + "numpy >= 1.22", so you can't help yourself and speed it up at all. Take or leave...

So I'm asking not "what happens" but "why should you check all 52 options" and "why do you need to check each one for that long time"? With a typical medium project, resolution time for each constraint decision easily goes up to 10-20 seconds per each override, and the task takes 100-200 seconds on fast SSD and fast computer. Is it that hard it can't be calculated in a part of a second? (of course when all deps infos are loaded into memory)

buriy avatar Jan 28 '22 18:01 buriy

I have no idea about the internals of the dependency resolver, so I cannot contribute to this discussion a lot. But:

you have only one python version installed so for example you shouldn't ever consider "python_version >= 3.10 alternative when you have python 3.8 installed.

Poetry is looking for a solution that is valid for the complete range of python version the projects aims to be compatible to. It doesn't care about the current Python version in use. This is different to e.g. pip and is intended.

finswimmer avatar Jan 28 '22 18:01 finswimmer

@finswimmer thanks. That's ok, you can still simplify it (Python >=3.8 & Python <3.10 &(( numpy >= 1.17.3 & Python>=3.7) | (Numpy >=1.19.3 & Python >= 3.9) | ...) ==> (numpy >= 1.19.3) or so, but what about the arch? Does it check for Windows packages around when setting up on Linux? Doesn't one worry that the packages versions will be incompatible on that arch? (I hope packages.lock won't fail if you have a cross-platform build! Will it?) (And for example redis is not available for Windows, for additional lulz, so if it aims to have a compatible build, it should break!).

buriy avatar Jan 28 '22 18:01 buriy

hi @buriy et al :) we've run into the same problems and I think we got something interesting for you: As we needed a quick remedy for this - we've forked poetry and made it consider only a subset of "target environment markers" which - as hinted / laid out in this thread - speeds up dependency resolution substantially. In our case, assuming a very specific environment for dependency resolution is totally fine as we're using container images for all our python development and shipping and thus control the environment using base image builds. this is the fork/branch (atm these changes are based on 1.1.8): https://github.com/Antaxify/poetry/tree/bshrk-1.1.8.

I hope i soon find the time

  • to provide some sample/fixture pyproject dependency structure

kazesberger avatar Feb 01 '22 08:02 kazesberger

@kazesberger and everyone, let's start with suggesting a better example -- which is rather short but takes for example an hour to resolve. 10 seconds to resolve won't attract enough attention. One hour should do. And it's fun.

It doesn't matter whether poetry considers a subset of markers, and it doesn't even matter whether it is consistent or not. But some brain power is really needed to be spent to fix the constraint solver. It's not that hard! I just don't have even a bit of time next month to fix this myself. But maybe I could consult/help someone else with the algorithms needed.

buriy avatar Feb 01 '22 19:02 buriy

But maybe I could consult/help someone else with the algorithms needed.

I'd really be curious what you think about my workaround (it's a very small change). please have a look, would really appreciate.

It doesn't matter whether poetry considers a subset of markers

trust me, you're 100% wrong here ;-) it does make a huge difference. like multiple hours down to <10mins for poetry lock

however my workaround represents a shortcut that will not suit every1. if you're collaborating with ppl on verious different devices with different architecture/os/whatever the lockfile is basically useless and needs to be regenerated by that/every user/dev.

kazesberger avatar Feb 03 '22 08:02 kazesberger

My team has this problem also, and it's hurting us a lot. In my case, each "retry" does take about 20sec, and it explores more than 64 retries for one of our simple packages.

I would love a hammer that allowed me to force a specific architecture and python version, since I'm deploying in a narrow environment.

I'd also like to understand a bit better what's going on. Many of the overrides it's using seem to be redundant, exploring say numpy (>=1.18.5), numpy (>=1.19.2) and numpy (>=1.21.0) in a single override. Why does it do that?

And, I have specified a specific version of numpy at my application level (1.19.1), so why doesn't it simply trust that specification? Indeed, that's the version it chooses after ~1hr of resolution anyway, so I'm puzzled why it works so hard at it.

Here's a snippet of output, showing that it's exploring each of 4 numpy bounds for each of three different pandas versions. This one ran through 64 retries before I Ctrl-C'd it.

   0: Retrying dependency resolution with the following overrides ({Package('pandas', '1.4.1'): {'numpy': <Dependency numpy (>=1.18.5)>}}).
   0: Retrying dependency resolution with the following overrides ({Package('pandas', '1.4.1'): {'numpy': <Dependency numpy (>=1.18.5)>}, Package('pandas', '1.4.0'): {'numpy': <Dependency numpy (>=1.18.5)>}}).
   0: Retrying dependency resolution with the following overrides ({Package('pandas', '1.4.1'): {'numpy': <Dependency numpy (>=1.18.5)>}, Package('pandas', '1.4.0'): {'numpy': <Dependency numpy (>=1.18.5)>}, Package('pandas', '1.3.5'): {'numpy': <Dependency numpy (>=1.17.3)>}}).
   0: Retrying dependency resolution with the following overrides ({Package('pandas', '1.4.1'): {'numpy': <Dependency numpy (>=1.18.5)>}, Package('pandas', '1.4.0'): {'numpy': <Dependency numpy (>=1.18.5)>}, Package('pandas', '1.3.5'): {'numpy': <Dependency numpy (>=1.19.2)>}}).
   0: Retrying dependency resolution with the following overrides ({Package('pandas', '1.4.1'): {'numpy': <Dependency numpy (>=1.18.5)>}, Package('pandas', '1.4.0'): {'numpy': <Dependency numpy (>=1.18.5)>}, Package('pandas', '1.3.5'): {'numpy': <Dependency numpy (>=1.20.0)>}}).
   0: Retrying dependency resolution with the following overrides ({Package('pandas', '1.4.1'): {'numpy': <Dependency numpy (>=1.18.5)>}, Package('pandas', '1.4.0'): {'numpy': <Dependency numpy (>=1.18.5)>}, Package('pandas', '1.3.5'): {'numpy': <Dependency numpy (>=1.21.0)>}}).
   0: Retrying dependency resolution with the following overrides ({Package('pandas', '1.4.1'): {'numpy': <Dependency numpy (>=1.18.5)>}, Package('pandas', '1.4.0'): {'numpy': <Dependency numpy (>=1.19.2)>}}).
   0: Retrying dependency resolution with the following overrides ({Package('pandas', '1.4.1'): {'numpy': <Dependency numpy (>=1.18.5)>}, Package('pandas', '1.4.0'): {'numpy': <Dependency numpy (>=1.19.2)>}, Package('pandas', '1.3.5'): {'numpy': <Dependency numpy (>=1.17.3)>}}).
   0: Retrying dependency resolution with the following overrides ({Package('pandas', '1.4.1'): {'numpy': <Dependency numpy (>=1.18.5)>}, Package('pandas', '1.4.0'): {'numpy': <Dependency numpy (>=1.19.2)>}, Package('pandas', '1.3.5'): {'numpy': <Dependency numpy (>=1.19.2)>}}).
   0: Retrying dependency resolution with the following overrides ({Package('pandas', '1.4.1'): {'numpy': <Dependency numpy (>=1.18.5)>}, Package('pandas', '1.4.0'): {'numpy': <Dependency numpy (>=1.19.2)>}, Package('pandas', '1.3.5'): {'numpy': <Dependency numpy (>=1.20.0)>}}).
   0: Retrying dependency resolution with the following overrides ({Package('pandas', '1.4.1'): {'numpy': <Dependency numpy (>=1.18.5)>}, Package('pandas', '1.4.0'): {'numpy': <Dependency numpy (>=1.19.2)>}, Package('pandas', '1.3.5'): {'numpy': <Dependency numpy (>=1.21.0)>}}).

Another attempt went through 93 overrides and succeeded; that try had some slightly different set of numpy versions it was considering, because of some tweak in pyproject.toml.

EDIT: In the above I didn't realize that I could pin the python version; doing so allowed me to eliminate one of the numpy versions, bringing the number of overrides down a lot. The remaining overrides are based on machine architecture, and I don't know how to control that parameter.

eekcoopuw avatar Feb 15 '22 01:02 eekcoopuw

I ran into the same issue. Newer versions of numpy and pandas seem to be the primary cause. In my current project the dependency resolution suddenly takes over an hour. With every new release of these librariers there is a new version to consider.

If I look at the runtime of my github actions of this project (https://github.com/mad-lab-fau/imucal/actions/workflows/test-and-lint.yml) you can see an increase in CI runtime from 4 min (last October) to over an hour (last week). I made no relevant changes that would explain this increase in runtime and this also matches my experience with other projects.

What works at the moment, is to change the minimal supported version of the library to 3.8. That drops the dependency resolution down to 4 minutes again. But this is not really an option...

EDIT: Just for further context: I was able to reduce the required time by chooseing Python >=3.7.1,<3.11 instead of ^3.7. But still Poetry tries 340 overrides.

AKuederle avatar Apr 12 '22 13:04 AKuederle

@AKuederle Thanks for this... almost had a brain aneurysm

jamespacileo avatar May 08 '22 16:05 jamespacileo

Alternative fix... keep Python to ^3.7...

and set pandas to ~1.3

Versions of Pandas of 1.4 onwards don't support Python 3.7... and lead you into the poetry dependency resolution black hole where time ends and nothing makes sense anymore.

For I got lost in the depths of the abyss and emerged to tell the tale... shall these words spread, inform and aid other unsuspecting travellers.

The abyss is there. Shrouded. Waiting for its next victim.

jamespacileo avatar May 08 '22 16:05 jamespacileo

@jamespacileo we were dealing with hours-long lock times as well, and replacing the requirement of pandas from ^x.y.z to ~x.y.z (i.e., only allow patches in z ) - reduced the lock time to 1.2 seconds so thanks for the idea :)

klayhb avatar Jul 13 '22 19:07 klayhb

the idea of speeding things up by trying to solve only for a narrowed environment is already proposed at #4952, which even has an MR at #4956.

I think this can be closed as a duplicate

dimbleby avatar Jul 17 '22 17:07 dimbleby

@jamespacileo thank you for the ladder bro, the abyss almost got me. For anyone who is experiencing a bunch overrides due to pandas/numpy, I locked my dependencies like so:

[tool.poetry.dependencies]
python = ">=3.7.1,<3.8"
pandas = "~1.3"

As my project requires python 3.7. Narrowing the range of acceptable python versions should alleviate this problem for most people.

data-djinn avatar Jul 18 '22 20:07 data-djinn

Hi all, just to share I seem to have found a way out of this abyss that allows supporting multiple Python versions from 3.7 to 3.9 at least.

In pyproject.toml I have:

[tool.poetry.dependencies]
python = ">=3.7.1,<3.10"
numpy = "*"
pandas = [
    {version="<1.4", python=">=3.7.1,<3.8"},
    {version="*", python=">=3.8,<3.10"}
]

This also only seems to work with poetry 1.2.0b3, not poetry 1.1 (which seems to omit including hashes for alternate pandas versions in the lock file).

alimanfoo avatar Jul 21 '22 13:07 alimanfoo

for some reason I was reminded of this one...

The original report didnt provide the things the issue template asked for, and of course lots of new versions of the mentioned packages have been published since then. So it is hard to compare apples with apples. poetry 1.1.12 was the latest release at the time, that much I am sure of.

Anyway as of today if we use these dependencies:

[tool.poetry.dependencies]
python = "^3.10"
numpy = "^1.26.3"
ipython = "^8.21.0"
jupyter = "^1.0.0"
opencv-python = "^4.9.0.80"
notebook = "^7.0.7"
pandas = "^2.2.0"
django = "~3.2"

as the best contemporary approximation of what is reported, then I see the following times for poetry lock:

  • poetry 1.1.12 with cache not populated: about four minutes (mostly spent compiling numpy)
  • poetry 1.1.12 with cache populated: 20 seconds ("with 24 overrides")
  • master branch with cache not populated: 20 seconds
  • master branch with cache populated: 5 seconds ("with 7 overrides")

which probably shows both that the ecosystem today is easier to solve, and also that there are several ways in which poetry has gotten faster over the last couple of years

more generally I think that https://github.com/python-poetry/poetry/issues/5121#issuecomment-1186572946 was right: the only useful actionable suggestion that I find in the discussion is that some folk would like the ability to solve for a narrowed environment, and so far as that goes this issue just duplicates others.

dimbleby avatar Feb 04 '24 21:02 dimbleby