Fairseq reinstalls pytorch

[nynMeta]

I have a source-compiled version of pytorch (1.12.0a0+git67ece03). When I tried to install fairseq, it uninstall the installed pytorch and install another version (1.12.1+cu102). Does anyone know how to resolve the issue? Here is the log of pip install -e . of Fairseq:

Obtaining file:///data/home/nyn/packages/fairseq
  Installing build dependencies ... done
  Checking if build backend supports build_editable ... done
  Getting requirements to build wheel ... done
  Installing backend dependencies ... done
  Preparing metadata (pyproject.toml) ... done
Requirement already satisfied: tqdm in /data/home/nyn/miniconda3/envs/myenv/lib/python3.8/site-packages (from fairseq==0.12.2) (4.64.0)
Requirement already satisfied: cffi in /data/home/nyn/miniconda3/envs/myenv/python3.8/site-packages (from fairseq==0.12.2) (1.15.1)
Requirement already satisfied: omegaconf<2.1 in /data/home/nyn/miniconda3/envs/myenv/lib/python3.8/site-packages (from fairseq==0.12.2) (2.0.6)
Requirement already satisfied: sacrebleu>=1.4.12 in /data/home/nyn/miniconda3/envs/myenv/lib/python3.8/site-packages (from fairseq==0.12.2) (2.2.0)
Requirement already satisfied: torchaudio>=0.8.0 in /data/home/nyn/miniconda3/envs/myenv/lib/python3.8/site-packages (from fairseq==0.12.2) (0.12.1)
Requirement already satisfied: bitarray in /data/home/nyn/miniconda3/envs/myenv/lib/python3.8/site-packages (from fairseq==0.12.2) (2.6.0)
Requirement already satisfied: hydra-core<1.1,>=1.0.7 in /data/home/nyn/miniconda3/envs/myenv/lib/python3.8/site-packages (from fairseq==0.12.2) (1.0.7)
Requirement already satisfied: regex in /data/home/nyn/miniconda3/envs/myenv/lib/python3.8/site-packages (from fairseq==0.12.2) (2022.7.25)
Requirement already satisfied: cython in /data/home/nyn/miniconda3/envs/myenv/lib/python3.8/site-packages (from fairseq==0.12.2) (0.29.32)
Requirement already satisfied: numpy in /data/home/nyn/miniconda3/envs/myenv/lib/python3.8/site-packages (from fairseq==0.12.2) (1.23.1)
Requirement already satisfied: torch in /data/home/nyn/miniconda3/envs/myenv/lib/python3.8/site-packages (from fairseq==0.12.2) (1.12.0a0+git67ece03)
Requirement already satisfied: antlr4-python3-runtime==4.8 in /data/home/nyn/miniconda3/envs/myenv/lib/python3.8/site-packages (from hydra-core<1.1,>=1.0.7->fairseq==0.12.2) (4.8)
Requirement already satisfied: importlib-resources in /data/home/nyn/miniconda3/envs/myenv/lib/python3.8/site-packages (from hydra-core<1.1,>=1.0.7->fairseq==0.12.2) (5.9.0)
Requirement already satisfied: typing-extensions in /data/home/nyn/miniconda3/envs/myenv/lib/python3.8/site-packages (from omegaconf<2.1->fairseq==0.12.2) (4.3.0)
Requirement already satisfied: PyYAML>=5.1.* in /data/home/nyn/miniconda3/envs/myenv/lib/python3.8/site-packages (from omegaconf<2.1->fairseq==0.12.2) (6.0)
Requirement already satisfied: colorama in /data/home/nyn/miniconda3/envs/myenv/lib/python3.8/site-packages (from sacrebleu>=1.4.12->fairseq==0.12.2) (0.4.5)
Requirement already satisfied: tabulate>=0.8.9 in /data/home/nyn/miniconda3/envs/myenv/lib/python3.8/site-packages (from sacrebleu>=1.4.12->fairseq==0.12.2) (0.8.10)
Requirement already satisfied: portalocker in /data/home/nyn/miniconda3/envs/myenv/lib/python3.8/site-packages (from sacrebleu>=1.4.12->fairseq==0.12.2) (2.5.1)
Requirement already satisfied: lxml in /data/home/nyn/miniconda3/envs/myenv/lib/python3.8/site-packages (from sacrebleu>=1.4.12->fairseq==0.12.2) (4.9.1
)
Collecting torch
  Using cached torch-1.12.1-cp38-cp38-manylinux1_x86_64.whl (776.3 MB)
Requirement already satisfied: pycparser in /data/home/nyn/miniconda3/envs/myenv/lib/python3.8/site-packages (from cffi->fairseq==0.12.2) (2.21)
Requirement already satisfied: zipp>=3.1.0 in /data/home/nyn/miniconda3/envs/myenv/lib/python3.8/site-packages (from importlib-resources->hydra-core<1.1,>=1.0.7->fairseq==0.12.2) (3.8.1)
Installing collected packages: torch, fairseq
  Attempting uninstall: torch
    Found existing installation: torch 1.12.0a0+git67ece03
    Uninstalling torch-1.12.0a0+git67ece03:
      Successfully uninstalled torch-1.12.0a0+git67ece03
  Running setup.py develop for fairseq
Successfully installed fairseq-0.12.2 torch-1.12.1

[erip]

I have just encoutered this as well. I managed to get this working by commenting out the torch and torchaudio dependencies in setup.py and this got me unblocked.

[mnoukhov]

This isn't a fairseq issue. This is regular pip behaviour see https://stackoverflow.com/questions/2861183/upgrade-package-without-upgrading-dependencies-using-pip

[erip]

No, it's not "regular pip behavior". The StackOverflow question you pointed us to is for upgrading packages which may require an upgrade of its dependencies. We're installing an editable version which should obey our currently installed dependencies. This is especially problematic since the version of pytorch isn't pinned (and therefore installing fairseq shouldn't do something dumb like installing pytorch built against a different version of CUDA...).

In short, it's definitively a bug. From the pypa documentation here (emphasis mine):

When your project is installed (e.g., using pip), all of the dependencies not already installed will be located (via PyPI), downloaded, built (if necessary), and installed

[nynMeta]

I have just encoutered this as well. I managed to get this working by commenting out the torch and torchaudio dependencies in setup.py and this got me unblocked.

Thanks for your tip. I got a work around by ignoring installing dependencies: pip install --no-deps -e . since I found I probably had all the required deps installed. However, I didn't do a rigorous test yet.

[weiyx16]

I think in another issue #4733 , this problem is fully discussed. For my case, adding --no-build-isolation will be ok.

[bl-sun]

I think in another issue #4733 , this problem is fully discussed. For my case, adding --no-build-isolation will be ok.

Why did I add this option and it still doesn't work? My order is pip install --editable ./ --no-build-isolation

[bl-sun]

I have just encoutered this as well. I managed to get this working by commenting out the torch and torchaudio dependencies in setup.py and this got me unblocked.

Thanks for your tip. I got a work around by ignoring installing dependencies: pip install --no-deps -e . since I found I probably had all the required deps installed. However, I didn't do a rigorous test yet.

Thank you very much, I tried your suggestion and it worked.