requirements.txt is overly restrictive

[bremoran]

Due to the extremely restrictive requirements.txt, local builds are forced for some packages. This means that some builds can fail. Would it be possible to change the requirements so that only the necessarily restrictive ones are restricted? For example, I installed whatever pip supplied for each of the dependencies, numpy, sympy, ortools, pandas, unicorn, and slothy appeared to work correctly.

sympy-1.13.3
unicorn-2.1.2
numpy-2.2.3
ortools-9.12.4544
pandas-2.2.3

This is as compared to the requirements.txt:

numpy==1.26.4
ortools==9.7.2996
pandas==2.1.1
sympy==1.12
unicorn==2.1.1

[hanno-becker]

@bremoran You are right that at present the requirements are very specific, and thereby restrictive.

One issue deserving particular emphasis is the OR-Tools version. We have observed severe performance regression from v9.8 onwards, which prevented us from upgrading so far, despite serious attempts to get to the bottom of this. We have not yet tested v9.12, though.

With v9.7 working best as far as we can tell so far, this also forces a particular Python version since v9.7 is no longer available in PyPI in newest Python versions. Hence the ultimate conclusion that you need a combination of (a) a pyenv for fixing the Python version, (b) a venv for fixing package installation.

You should not force those installations in your global Python environment -- always use a venv.

Does working in a pyenv+venv resolve your concern?

[bremoran]

Of course, using a venv is standard practice and outlined in your readme & tutorial. That said, since you have seen performance regression in ortools, I would have expected a restriction on the non-performant version, for example:

ortools>=9.7,<9.8
numpy
pandas
sympy
unicorn

[hanno-becker]

I do remember there once was an issue with numpy which forced a version <2, but I don't recall the details. I don't recall reasons why we needed to lock the other packages -- I think it was done as part of the artifact preparation for CHES, to ensure reproducibility. But since the artifact is locked down and archived now, we can change this in main.

My recommendation would be to open a PR where we try to merely lock ortools, keep the rest unspecified, and just see what happens. @mkannwischer @dop-amin please chime in.

[mkannwischer]

For development, I value having the same versions across multiple developers more than supporting many versions. It saves us one variable when debugging.

For users I can understand that this is inconvenient. But if we do want to support more versions, we will need more extensive CI to make sure everything keeps working with the supported versions. I dont think that is feasible with our current resources.

I am not in favour of just removing the version pinning.

Now, if the dependencies trigger local builds - that is a problem. I'd want to address that by updating the respective dependencies.