Requirements in different places are conflicting each other

[vnlitvinov]

System information

• Modin version (modin.__version__): 653d5b235ffe373b1416fc8049c1080db89bc156

Describe the problem

We have a few places specifying requirements for Modin: setup.py, requirements.txt (and a few other .txt files) and a few .yaml files, as well as some places in code where we check the versions in runtime.

One certain conflict for now is setup.py requires pyarrow=1.0 while OmniSci requires pyarrow=2.0.

We need to make sure all our requirements are using some single source of truth.

It feels that using requirements.txt files (one per extra feature) is probably the best option as truth source as they're the easiest to parse and utilize, so we should:

1. make the unified list of requirements
2. change all the places where we specify the requirements so they are taking the requirements from the single source, so it never goes out of sync again

[vnlitvinov]

Food for thought: current requirements on distributed prevent us from making a release for Python 3.9 at least on conda-forge, ref: https://github.com/conda-forge/modin-feedstock/pull/17

[devin-petersohn]

@vnlitvinov Is this still an issue?

[Garra1980]

I believe we still want to have single source of requirements

[vnlitvinov]

Idea: instead of generating things out of one source of truth we can instead make a script (running during CI) which would ensure our requirements are coherent at all times. Simplest implementation would be parsing all the requirements files and checking version requirements for each component individually - should be rather easy to implement.

[RehanSD]

@vnlitvinov has this issue been resolved? I believe we've synchronized a lot of our requirements, but are we still hoping to have one source of truth for requirements instead of multiple files?

[rootsmusic]

I second @vnlitvinov's idea for a script to run during CI. At the very least, please document dependencies on the installation page.