Document how to use from Docker?

[9876691]

Could you add some notes to the Docker section on how to run inference?

I tried

docker run -it --gpus all mosaicml/llm-foundry:2.0.1_cu118-latest bash

But there doesn't seem to be a scripts folder, so I wasn't sure how to proceed.

Ideally I'd like instructions on how to run inference and training with docker on 7b.

Thanks

[srivassid]

You have to clone the repo and run pip install .[gpu] to proceed further

[9876691]

So now I get an error when doing pip install.

(llmfoundry-venv) root@cd2c34937d93:/home/mosaicml/llm-foundry# pip install -e ".[gpu]"
Obtaining file:///home/mosaicml/llm-foundry
  Installing build dependencies ... done
  Checking if build backend supports build_editable ... done
  Getting requirements to build editable ... done
  Installing backend dependencies ... done
  Preparing editable metadata (pyproject.toml) ... done
Collecting triton-pre-mlir@ git+https://github.com/vchiley/triton.git@triton_pre_mlir_sm90#subdirectory=python
  Cloning https://github.com/vchiley/triton.git (to revision triton_pre_mlir_sm90) to /tmp/pip-install-hqwcutn6/triton-pre-mlir_c3f21e6bfcb74eb9a99c530ee6a6153e
  Running command git clone --filter=blob:none --quiet https://github.com/vchiley/triton.git /tmp/pip-install-hqwcutn6/triton-pre-mlir_c3f21e6bfcb74eb9a99c530ee6a6153e
  Running command git checkout -b triton_pre_mlir_sm90 --track origin/triton_pre_mlir_sm90
  Switched to a new branch 'triton_pre_mlir_sm90'
  branch 'triton_pre_mlir_sm90' set up to track 'origin/triton_pre_mlir_sm90'.
  Resolved https://github.com/vchiley/triton.git to commit 48b1cc9ff8b1f506ac32f2124471e2582875c008
  Running command git submodule update --init --recursive -q
  Preparing metadata (setup.py) ... done
Collecting xentropy-cuda-lib@ git+https://github.com/HazyResearch/[email protected]#subdirectory=csrc/xentropy
  Cloning https://github.com/HazyResearch/flash-attention.git (to revision v1.0.3) to /tmp/pip-install-hqwcutn6/xentropy-cuda-lib_c6bebc961fce437688c984b037ace232
  Running command git clone --filter=blob:none --quiet https://github.com/HazyResearch/flash-attention.git /tmp/pip-install-hqwcutn6/xentropy-cuda-lib_c6bebc961fce437688c984b037ace232
  Running command git checkout -q 67ef5d28df71d395bc16787b31e08ea1afbe4178
  Resolved https://github.com/HazyResearch/flash-attention.git to commit 67ef5d28df71d395bc16787b31e08ea1afbe4178
  Running command git submodule update --init --recursive -q
  Preparing metadata (setup.py) ... error
  error: subprocess-exited-with-error
  
  × python setup.py egg_info did not run successfully.
  │ exit code: 1
  ╰─> [6 lines of output]
      Traceback (most recent call last):
        File "<string>", line 2, in <module>
        File "<pip-setuptools-caller>", line 34, in <module>
        File "/tmp/pip-install-hqwcutn6/xentropy-cuda-lib_c6bebc961fce437688c984b037ace232/csrc/xentropy/setup.py", line 5, in <module>
          from packaging.version import parse, Version
      ModuleNotFoundError: No module named 'packaging'
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
error: metadata-generation-failed

× Encountered error while generating package metadata.
╰─> See above for output.

note: This is an issue with the package mentioned above, not pip.
hint: See above for details.

[notice] A new release of pip is available: 23.0.1 -> 23.1.2
[notice] To update, run: pip install --upgrade pip

[srivassid]

can you try installing package packing and running again?

[Symbolk]

can you try installing package packing and running again?

Same problem but pip install packaging makes no difference.

[Symbolk]

So now I get an error when doing pip install.

(llmfoundry-venv) root@cd2c34937d93:/home/mosaicml/llm-foundry# pip install -e ".[gpu]"
Obtaining file:///home/mosaicml/llm-foundry
  Installing build dependencies ... done
  Checking if build backend supports build_editable ... done
  Getting requirements to build editable ... done
  Installing backend dependencies ... done
  Preparing editable metadata (pyproject.toml) ... done
Collecting triton-pre-mlir@ git+https://github.com/vchiley/triton.git@triton_pre_mlir_sm90#subdirectory=python
  Cloning https://github.com/vchiley/triton.git (to revision triton_pre_mlir_sm90) to /tmp/pip-install-hqwcutn6/triton-pre-mlir_c3f21e6bfcb74eb9a99c530ee6a6153e
  Running command git clone --filter=blob:none --quiet https://github.com/vchiley/triton.git /tmp/pip-install-hqwcutn6/triton-pre-mlir_c3f21e6bfcb74eb9a99c530ee6a6153e
  Running command git checkout -b triton_pre_mlir_sm90 --track origin/triton_pre_mlir_sm90
  Switched to a new branch 'triton_pre_mlir_sm90'
  branch 'triton_pre_mlir_sm90' set up to track 'origin/triton_pre_mlir_sm90'.
  Resolved https://github.com/vchiley/triton.git to commit 48b1cc9ff8b1f506ac32f2124471e2582875c008
  Running command git submodule update --init --recursive -q
  Preparing metadata (setup.py) ... done
Collecting xentropy-cuda-lib@ git+https://github.com/HazyResearch/[email protected]#subdirectory=csrc/xentropy
  Cloning https://github.com/HazyResearch/flash-attention.git (to revision v1.0.3) to /tmp/pip-install-hqwcutn6/xentropy-cuda-lib_c6bebc961fce437688c984b037ace232
  Running command git clone --filter=blob:none --quiet https://github.com/HazyResearch/flash-attention.git /tmp/pip-install-hqwcutn6/xentropy-cuda-lib_c6bebc961fce437688c984b037ace232
  Running command git checkout -q 67ef5d28df71d395bc16787b31e08ea1afbe4178
  Resolved https://github.com/HazyResearch/flash-attention.git to commit 67ef5d28df71d395bc16787b31e08ea1afbe4178
  Running command git submodule update --init --recursive -q
  Preparing metadata (setup.py) ... error
  error: subprocess-exited-with-error
  
  × python setup.py egg_info did not run successfully.
  │ exit code: 1
  ╰─> [6 lines of output]
      Traceback (most recent call last):
        File "<string>", line 2, in <module>
        File "<pip-setuptools-caller>", line 34, in <module>
        File "/tmp/pip-install-hqwcutn6/xentropy-cuda-lib_c6bebc961fce437688c984b037ace232/csrc/xentropy/setup.py", line 5, in <module>
          from packaging.version import parse, Version
      ModuleNotFoundError: No module named 'packaging'
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
error: metadata-generation-failed

× Encountered error while generating package metadata.
╰─> See above for output.

note: This is an issue with the package mentioned above, not pip.
hint: See above for details.

[notice] A new release of pip is available: 23.0.1 -> 23.1.2
[notice] To update, run: pip install --upgrade pip

Hi, I am stuck in the same problem, any progress on this?

[alextrott16]

I think we've made some updates to our dependencies, so I'm wondering if people are still running into this issue?

[Symbolk]

I managed to manually install triton@48b1cc9ff8b1f506ac32f2124471e2582875c008, and then it works.

[hanlint]

Closing as stale -- please re-open if there are more issues. We've updated a lot of our install documentation, so hopefully this is not an issue anymore.