Error installing on windows

[NicolasMejiaPetit]

NameError: name 'nvcc_cuda_version' is not defined. Did you mean: 'cuda_version'?

This was a simple fix I defined the cuda version in line 268 of setup .py and installed with 'pip install .'

so change line 268 from: cuda_version = str(nvcc_cuda_version) to: cuda_version = str(12.1)

If this problem is common amongst windows users you could add a precheck for os version, and if windows, allow user to set cuda version via prompt.

Although i still have a error building the wheels further down the line:

  copying vllm\transformers_utils\tokenizers\__init__.py -> build\lib.win-amd64-cpython-311\vllm\transformers_utils\tokenizers
  copying vllm\py.typed -> build\lib.win-amd64-cpython-311\vllm
  running build_ext
  C:\Users\PC\AppData\Local\Temp\pip-build-env-8j8g0uyh\overlay\Lib\site-packages\torch\utils\cpp_extension.py:383: UserWarning: Error checking compiler version for cl: [WinError 2] The system cannot find the file specified
    warnings.warn(f'Error checking compiler version for {compiler}: {error}')
  Traceback (most recent call last):
    File "C:\Python311\Lib\site-packages\pip\_vendor\pyproject_hooks\_in_process\_in_process.py", line 353, in <module>
      main()
    File "C:\Python311\Lib\site-packages\pip\_vendor\pyproject_hooks\_in_process\_in_process.py", line 335, in main
      json_out['return_val'] = hook(**hook_input['kwargs'])
                               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "C:\Python311\Lib\site-packages\pip\_vendor\pyproject_hooks\_in_process\_in_process.py", line 251, in build_wheel
      return _build_backend().build_wheel(wheel_directory, config_settings,
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "C:\Users\PC\AppData\Local\Temp\pip-build-env-8j8g0uyh\overlay\Lib\site-packages\setuptools\build_meta.py", line 404, in build_wheel
      return self._build_with_temp_dir(
             ^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "C:\Users\PC\AppData\Local\Temp\pip-build-env-8j8g0uyh\overlay\Lib\site-packages\setuptools\build_meta.py", line 389, in _build_with_temp_dir
      self.run_setup()
    File "C:\Users\PC\AppData\Local\Temp\pip-build-env-8j8g0uyh\overlay\Lib\site-packages\setuptools\build_meta.py", line 311, in run_setup
      exec(code, locals())
    File "<string>", line 296, in <module>
    File "C:\Users\PC\AppData\Local\Temp\pip-build-env-8j8g0uyh\overlay\Lib\site-packages\setuptools\__init__.py", line 103, in setup
      return distutils.core.setup(**attrs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "C:\Users\PC\AppData\Local\Temp\pip-build-env-8j8g0uyh\overlay\Lib\site-packages\setuptools\_distutils\core.py", line 185, in setup
      return run_commands(dist)
             ^^^^^^^^^^^^^^^^^^
    File "C:\Users\PC\AppData\Local\Temp\pip-build-env-8j8g0uyh\overlay\Lib\site-packages\setuptools\_distutils\core.py", line 201, in run_commands
      dist.run_commands()
    File "C:\Users\PC\AppData\Local\Temp\pip-build-env-8j8g0uyh\overlay\Lib\site-packages\setuptools\_distutils\dist.py", line 969, in run_commands
      self.run_command(cmd)
    File "C:\Users\PC\AppData\Local\Temp\pip-build-env-8j8g0uyh\overlay\Lib\site-packages\setuptools\dist.py", line 963, in run_command
      super().run_command(command)
    File "C:\Users\PC\AppData\Local\Temp\pip-build-env-8j8g0uyh\overlay\Lib\site-packages\setuptools\_distutils\dist.py", line 988, in run_command
      cmd_obj.run()
    File "C:\Users\PC\AppData\Local\Temp\pip-build-env-8j8g0uyh\overlay\Lib\site-packages\wheel\bdist_wheel.py", line 368, in run
      self.run_command("build")
    File "C:\Users\PC\AppData\Local\Temp\pip-build-env-8j8g0uyh\overlay\Lib\site-packages\setuptools\_distutils\cmd.py", line 318, in run_command
      self.distribution.run_command(command)
    File "C:\Users\PC\AppData\Local\Temp\pip-build-env-8j8g0uyh\overlay\Lib\site-packages\setuptools\dist.py", line 963, in run_command
      super().run_command(command)
    File "C:\Users\PC\AppData\Local\Temp\pip-build-env-8j8g0uyh\overlay\Lib\site-packages\setuptools\_distutils\dist.py", line 988, in run_command
      cmd_obj.run()
    File "C:\Users\PC\AppData\Local\Temp\pip-build-env-8j8g0uyh\overlay\Lib\site-packages\setuptools\_distutils\command\build.py", line 131, in run
      self.run_command(cmd_name)
    File "C:\Users\PC\AppData\Local\Temp\pip-build-env-8j8g0uyh\overlay\Lib\site-packages\setuptools\_distutils\cmd.py", line 318, in run_command
      self.distribution.run_command(command)
    File "C:\Users\PC\AppData\Local\Temp\pip-build-env-8j8g0uyh\overlay\Lib\site-packages\setuptools\dist.py", line 963, in run_command
      super().run_command(command)
    File "C:\Users\PC\AppData\Local\Temp\pip-build-env-8j8g0uyh\overlay\Lib\site-packages\setuptools\_distutils\dist.py", line 988, in run_command
      cmd_obj.run()
    File "C:\Users\PC\AppData\Local\Temp\pip-build-env-8j8g0uyh\overlay\Lib\site-packages\setuptools\command\build_ext.py", line 88, in run
      _build_ext.run(self)
    File "C:\Users\PC\AppData\Local\Temp\pip-build-env-8j8g0uyh\overlay\Lib\site-packages\setuptools\_distutils\command\build_ext.py", line 345, in run
      self.build_extensions()
    File "C:\Users\PC\AppData\Local\Temp\pip-build-env-8j8g0uyh\overlay\Lib\site-packages\torch\utils\cpp_extension.py", line 525, in build_extensions
      _check_cuda_version(compiler_name, compiler_version)
    File "C:\Users\PC\AppData\Local\Temp\pip-build-env-8j8g0uyh\overlay\Lib\site-packages\torch\utils\cpp_extension.py", line 407, in _check_cuda_version
      torch_cuda_version = packaging.version.parse(torch.version.cuda)
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "C:\Users\PC\AppData\Local\Temp\pip-build-env-8j8g0uyh\overlay\Lib\site-packages\pkg_resources\_vendor\packaging\version.py", line 52, in parse
      return Version(version)
             ^^^^^^^^^^^^^^^^
    File "C:\Users\PC\AppData\Local\Temp\pip-build-env-8j8g0uyh\overlay\Lib\site-packages\pkg_resources\_vendor\packaging\version.py", line 196, in __init__
      match = self._regex.search(version)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  TypeError: expected string or bytes-like object, got 'NoneType'
  [end of output]

note: This error originates from a subprocess, and is likely not a problem with pip. ERROR: Failed building wheel for vllm Building wheel for quantile-python (setup.py) ... done Created wheel for quantile-python: filename=quantile_python-1.1-py3-none-any.whl size=3452 sha256=14bb55c2f1ae4594a664c5ad99d3ed31e792f9eae5939d17af0372a1bb6bd6ae Stored in directory: c:\users\pc\appdata\local\pip\cache\wheels\67\a2\17\29e7169adf03a7e44b922abb6a42c2c1b0fda11f7bfbdb24a2 Successfully built quantile-python Failed to build vllm ERROR: Could not build wheels for vllm, which is required to install pyproject.toml-based projects

[notice] A new release of pip is available: 23.1.2 -> 23.3.2 [notice] To update, run: python.exe -m pip install --upgrade pip

C:\Users\PC\Documents\vllm-main>

The error remains in text gerneration web ui env

  copying vllm\py.typed -> build\lib.win-amd64-cpython-311\vllm
  running build_ext
  C:\Users\PC\Documents\NEWGEN\text-generation-webui-main\installer_files\pip-build-env-x2t9cg8m\overlay\Lib\site-packages\torch\utils\cpp_extension.py:383: UserWarning: Error checking compiler version for cl: [WinError 2] The system cannot find the file specified
    warnings.warn(f'Error checking compiler version for {compiler}: {error}')
  error: [WinError 2] The system cannot find the file specified
  [end of output]

note: This error originates from a subprocess, and is likely not a problem with pip. ERROR: Failed building wheel for vllm Failed to build vllm ERROR: Could not build wheels for vllm, which is required to install pyproject.toml-based projects

(C:\Users\PC\Documents\NEWGEN\text-generation-webui-main\installer_files\env) C:\Users\PC\Documents\vllm-main>

[GDTyron]

I had the same issue. It originates from the fact that torch.version.cuda is an empty string if you haven't installed CUDA or GPU acceleration is disabled in your torch version. I'm guessing vllm doesn't support running on CPU, so even if you install CUDA and a CPU based torch version, you still can't install vllm.

This is my experience, I am more than happy to be corrected by others about this issue.

[NicolasMejiaPetit]

Strange what was your solution, like what did you install for it to work?

I have a 3090 so it should be detecting my gpu, and I'm able to use the same environment to fine tune models.

I got it working on wsl, but my entire work load is on windows so switching back and forth is annoying.

[GDTyron]

Linux is listed as a requirement for vllm in the documentation. So I'm guessing WSL is the best way to do things.

As a sidenote, I am trying to install vllm without a GPU and got other errors. Do you mind telling me the version of PyTorch you're using?

[vlserov]

By default, on Windows installed PyTorch CPU version. For Lunix, it the PyTorch with CUDA 12.1. So I've tried to specify default index by: pip install . --index-url https://download.pytorch.org/whl/cu121 --extra-index-url https://pypi.org/simple.

[WAS-PlaiLabs]

I have CUDA installed, however this error persists. This is unfortunate, as now llama-index is inoperable on Windows.

[elkay]

Same as previous person, except on Amazon Linux.

CUDA is installed correctly and available.

(textgen) [ec2-user@ip-10-0-1-88 vllm]$ python3 -c "import torch; print('Using device: ', torch.device('cuda' if torch.cuda.is_available() else 'cpu'))";
Using device:  cuda

[thesues]

I was able to INSTALL vllm on ms windows(not WSL). But I still meet runtime errors which be hard to resolve. this is what I did:

1. install nvidia tool kit 12.1
2. do not call 'pip install vllm'. because pip will install pytorch(CPU version) and you know vllm must have CUDA now.

pip3 install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu121

3. git clone this vllm repo
4. build and install vllm from source

python setup.py install

But you will still meet some compiling errors. such as as nvcc do not know asm or std17 can not work. I removed awq related code. so all kernel src could be compiled and installed.

5. However, in the runtime. vllm requires xformers, and xformers requires openai triton, and triton does not support windows right now. Even if we can hack the code to remove formers dependency, it is not a good idea.

[NicolasMejiaPetit]

Need Triton on native windows? IGU don't worry. @thesues I Previously made these instructions for installing Unsloth AI on windows, which has the same requirements. Here ya go:

Native Windows Install (No WSL)

Requirements

• Cuda 12.1 (Untested on 11.8 although it probably would work)
• Python 3.11
• Visual Studio 2022 build tools [^1^][1]

Installing Visual Studio 2022 Build Tools

1. First, you must install the installer.
2. Open Visual Studio installer and select 'Desktop Development with C++' under the "Workload" tab.
3. Under the Individual Components Tab, search for these optional components if not automatically selected: "MSVC v143-VS 2022 C++ x64/x86 build tools, Windows 11 SDK, C++ CMake tools for windows, Testing tools core features-Build Tools, C++ AddressSanitizer".
4. Hit install and close Visual Studio 2022.

Installing Premade Wheels and Triton Library

Wheels and Library

This is a link to my google drive, where i uploaded my wheels, and the triton libraries you must add to your environment path, made for python 3.11, and cuda 12.1

Note: I have made pre-made wheel Deepspeed (These wheels are experimental wheels, just because you have them installed does not mean all the calls made work on non-unix based OS, lucky for us Single GPU trainers, it does work on windows even when going into shared memory. I did not make the Triton windows wheels or windows library for Triton, they were made by wkpark on GitHub).

Installing Triton

1. The first package you will need to install is Triton, as Deepspeed depends on it.
2. In order to install Triton, you will need the Triton Libraries. Unzip the Triton libraries to your C drive like so 'C:\llvm-5e5a22ca-windows-x64'.
3. Add The Bin and lib to your environment variables (same way you did when installing CUDA) so 'C:\llvm-5e5a22ca-windows-x64\bin' and 'C:\llvm-5e5a22ca-windows-x64\lib' to your system path inside the environment variables page.
4. Now you may install the wheels. Download the wheels and run 'pip install triton-2.1.0-cp311-cp311-win_amd64.whl'.

Installing Deepspeed

1. Download the package and run 'pip install deepspeed-0.13.1+unknown-py3-none-any.whl'.

Installing Latest BitsAndBytes Release Windows Package

1. Download the package and run 'pip install bitsandbytes-0.43.0.dev0-cp311-cp311-win_amd64.whl'.

Install Xformers

1. Run 'pip install xformers'. Note: you might get issues running training related to Xformers. If you do, run 'pip uninstall xformers' then 'pip install xformers'.

There might be another error present with installing xformers, if it requires you to download the linux package

1. clone the repository
2. Edit the project.toml file to use xformers-0.0.24-cp311-cp311-win_amd64.whl on the version you are installing. instead of the linux xformers
3. Should look like this: '"xformers @ https://download.pytorch.org/whl/cu121/xformers-0.0.24-cp311-cp311-win_amd64.whl ; python_version=='3.11'",'

Windows install instructions made by Nicolas Mejia Petit

[elkay]

FYI this process installs a non-CUDA version of torch by default. If you have and want to use CUDA, you'll need to finish up with uninstalling torch and installing an appropriate CUDA version for whatever your CUDA version is. You might also need to include the CUDA version of the whl during install, per the same as xformers instructions above.

pip uninstall torch
pip install torch==2.1.2+cu118 --index-url https://download.pytorch.org/whl/cu118

I would also suggest doing all of this entire install process in an isolated environment so you don't muck up your default python install.

conda create -n vllm python=3.11
activate vllm

[NicolasMejiaPetit]

I was able to INSTALL vllm on ms windows(not WSL). But I still meet runtime errors which be hard to resolve. this is what I did:

1. install nvidia tool kit 12.1
2. do not call 'pip install vllm'. because pip will install pytorch(CPU version) and you know vllm must have CUDA now.

pip3 install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu121

3. git clone this vllm repo
4. build and install vllm from source

python setup.py install

But you will still meet some compiling errors. such as as nvcc do not know asm or std17 can not work. I removed awq related code. so all kernel src could be compiled and installed.

5. However, in the runtime. vllm requires xformers, and xformers requires openai triton, and triton does not support windows right now. Even if we can hack the code to remove formers dependency, it is not a good idea.

@thesues Could you share with me the edited setup.py (or any other files you might’ve changed) so I could test it out with my window environment, I have deep speed and triton installed. (The wheels and instructions are above if you want to install them as well) but I’m getting those errors you were referring to.

[hmellor]

Closing as vLLM is only supported in Windows via WSL

[WAS-PlaiLabs]

I have this installed on windows now, not sure what the steps were to get it to, but it works under normal windows python venv for llama-index. Not sure if that's the brunt of it's functionality and if it's fully working though. But llama-index doesn't complain and it all runs.

[NicolasMejiaPetit]

@WAS-PlaiLabs

What did you have to change locally in order for it to install?

[WAS-PlaiLabs]

@WAS-PlaiLabs

What did you have to change locally in order for it to install?

I'm not positive. I think it's cause I updated build tools. But when I came across this thread I was trying so many potential fixes. Not much help. I do think though that now that it is installed there should be a pre-built wheel somewhere?

[NicolasMejiaPetit]

@WAS-PlaiLabs @thesues Yeah, look up .whl in the search bar of the vllm folder, it should be in like a build folder or something. Did you do any modifications to the setup.py or anything to install it? And if you could please share me the wheel I’m seriously begging I can’t get it working on my end. Or the entire vllm folder you have with the modifications if you don’t remember the stuff you changed please :) I really need this. You could use google drive or drop box or anything idc.