FileNotFoundError: [Errno 2] No such file or directory: 'megatron/core/requirements.txt'

[make1986]

Describe the bug

Can't install nemo-toolkit

Steps/Code to reproduce bug Tried to do it through pip and source and getting:

error: subprocess-exited-with-error
  
  × Getting requirements to build wheel did not run successfully.
  │ exit code: 1
  ╰─> [18 lines of output]
      Traceback (most recent call last):
        File "/usr/local/lib/python3.8/dist-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 353, in <module>
          main()
        File "/usr/local/lib/python3.8/dist-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 335, in main
          json_out['return_val'] = hook(**hook_input['kwargs'])
        File "/usr/local/lib/python3.8/dist-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 118, in get_requires_for_build_wheel
          return hook(config_settings)
        File "/tmp/pip-build-env-fgqdh0em/overlay/lib/python3.8/site-packages/setuptools/build_meta.py", line 325, in get_requires_for_build_wheel
          return self._get_build_requires(config_settings, requirements=['wheel'])
        File "/tmp/pip-build-env-fgqdh0em/overlay/lib/python3.8/site-packages/setuptools/build_meta.py", line 295, in _get_build_requires
          self.run_setup()
        File "/tmp/pip-build-env-fgqdh0em/overlay/lib/python3.8/site-packages/setuptools/build_meta.py", line 480, in run_setup
          super(_BuildMetaLegacyBackend, self).run_setup(setup_script=setup_script)
        File "/tmp/pip-build-env-fgqdh0em/overlay/lib/python3.8/site-packages/setuptools/build_meta.py", line 311, in run_setup
          exec(code, locals())
        File "<string>", line 52, in <module>
        File "<string>", line 45, in req_file
      FileNotFoundError: [Errno 2] No such file or directory: 'megatron/core/requirements.txt'
      [end of output]

Environment overview (please complete the following information)

• Environment location: [Bare-metal, Docker, Cloud(specify cloud provider - AWS, Azure, GCP, Collab)]
• Method of NeMo install: [pip install or from source]. Please specify exact commands you used to install.
• If method of install is [Docker], provide docker pull & docker run commands used

Environment details

• Ubuntu 20.04
• python 3.8, 3.9, 3.10, 3.11

Additional context RTX 3090 CUDA 12.2

[t-hem]

Having same issue when installing from pip:

Collecting megatron-core==0.4.0 (from nemo_toolkit[all])
  Downloading megatron_core-0.4.0.tar.gz (154 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 154.8/154.8 kB 8.7 MB/s eta 0:00:00
  Installing build dependencies ... done
  Getting requirements to build wheel ... error
  error: subprocess-exited-with-error
  
  × Getting requirements to build wheel did not run successfully.
  │ exit code: 1
  ╰─> [21 lines of output]
      Traceback (most recent call last):
        File "/home/thomas/AI-Project/.conda/lib/python3.11/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 353, in <module>
          main()
        File "/home/thomas/AI-Project/.conda/lib/python3.11/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 335, in main
          json_out['return_val'] = hook(**hook_input['kwargs'])
                                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/home/thomas/AI-Project/.conda/lib/python3.11/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 118, in get_requires_for_build_wheel
          return hook(config_settings)
                 ^^^^^^^^^^^^^^^^^^^^^
        File "/tmp/pip-build-env-m4c0trdg/overlay/lib/python3.11/site-packages/setuptools/build_meta.py", line 325, in get_requires_for_build_wheel
          return self._get_build_requires(config_settings, requirements=['wheel'])
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/tmp/pip-build-env-m4c0trdg/overlay/lib/python3.11/site-packages/setuptools/build_meta.py", line 295, in _get_build_requires
          self.run_setup()
        File "/tmp/pip-build-env-m4c0trdg/overlay/lib/python3.11/site-packages/setuptools/build_meta.py", line 480, in run_setup
          super(_BuildMetaLegacyBackend, self).run_setup(setup_script=setup_script)
        File "/tmp/pip-build-env-m4c0trdg/overlay/lib/python3.11/site-packages/setuptools/build_meta.py", line 311, in run_setup
          exec(code, locals())
        File "<string>", line 52, in <module>
        File "<string>", line 45, in req_file
      FileNotFoundError: [Errno 2] No such file or directory: 'megatron/core/requirements.txt'
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
error: subprocess-exited-with-error

× Getting requirements to build wheel did not run successfully.
│ exit code: 1
╰─> See above for output.

note: This error originates from a subprocess, and is likely not a problem with pip.

Environment overview (please complete the following information)

Environment location: local
Method of NeMo install:  pip install nemo_toolkit['all']

Environment details

Ubuntu 22.04 LTS
python 3.11

Additional context RTX 4080M CUDA 12.2

[nahue-passano]

Hey everyone, I stumbled upon the same issue. In this commit c67243521b048e7ce1446a0d45a0b6e262528e73 they merged a change updating the megatron version from 0.3.0 to 0.4.0, which seems to be the root of the problem.

I managed to resolve it by installing an earlier release v1.21.0 instead of the current one v1.22.0.

To install from the previous release, use the following pip command:

pip install nemo_toolkit[all]==1.21

I hope this helps!

[github-actions[bot]]

This issue is stale because it has been open for 30 days with no activity. Remove stale label or comment or this will be closed in 7 days.

[iambsk]

This issue still exists when following the command: pip install git+https://github.com/NVIDIA/[email protected]#egg=nemo_toolkit[all] as instructed here: nvidia/canary-1b

[stph-laagencia]

This issue still exists when following the command: pip install git+https://github.com/NVIDIA/[email protected]#egg=nemo_toolkit[all] as instructed here: nvidia/canary-1b

It's true I have the same issue following the nvidia/canary-1b instructions

[krishnacpuvvada]

This issue still exists when following the command: pip install git+https://github.com/NVIDIA/[email protected]#egg=nemo_toolkit[all] as instructed here: nvidia/canary-1b

It's true I have the same issue following the nvidia/canary-1b instructions

can you try pip install git+https://github.com/NVIDIA/[email protected]#egg=nemo_toolkit[asr] instead of [all] for canary

[github-actions[bot]]

This issue is stale because it has been open for 30 days with no activity. Remove stale label or comment or this will be closed in 7 days.

[github-actions[bot]]

This issue was closed because it has been inactive for 7 days since being marked as stale.