MiniCPM-o icon indicating copy to clipboard operation
MiniCPM-o copied to clipboard

Published 20 hours ago • verified publisher icon OpenBMB

方便提供openai_api接口吗，这样我就可以做成服务调用了，VLLM就更好了

Open yanghp86 opened this issue 10 months ago • 7 comments

方便提供openai_api接口吗，这样我就可以做成服务调用了，VLLM就更好了，听说这周就可以实现吗？

yanghp86 avatar Apr 18 '24 06:04 yanghp86

您好，感谢关注！我们下周会提供API接口，供大家调用。VLLM我们已经向官方提交了PR，应该近几天审核通过后会正式提供出来。

yaoyuanTHU avatar Apr 18 '24 16:04 yaoyuanTHU

已支持vLLM 推理，欢迎试用～

iceflame89 avatar Apr 24 '24 12:04 iceflame89

已支持vLLM 推理，欢迎试用～

尝试使用 vllm 报错了

Obtaining file:///data/workspace/MiniCPM/vllm
  Installing build dependencies ... done
  Checking if build backend supports build_editable ... done
  Getting requirements to build editable ... done
  Preparing editable metadata (pyproject.toml) ... done
Requirement already satisfied: cmake>=3.21 in /root/anaconda3/envs/vllm/lib/python3.9/site-packages (from vllm==0.4.1+cu122) (3.29.2)
Requirement already satisfied: ninja in /root/anaconda3/envs/vllm/lib/python3.9/site-packages (from vllm==0.4.1+cu122) (1.11.1.1)
Requirement already satisfied: psutil in /root/anaconda3/envs/vllm/lib/python3.9/site-packages (from vllm==0.4.1+cu122) (5.9.8)
Requirement already satisfied: sentencepiece in /root/anaconda3/envs/vllm/lib/python3.9/site-packages (from vllm==0.4.1+cu122) (0.2.0)
Requirement already satisfied: numpy in /root/anaconda3/envs/vllm/lib/python3.9/site-packages (from vllm==0.4.1+cu122) (1.26.4)
Requirement already satisfied: requests in /root/anaconda3/envs/vllm/lib/python3.9/site-packages (from vllm==0.4.1+cu122) (2.31.0)
Requirement already satisfied: py-cpuinfo in /root/anaconda3/envs/vllm/lib/python3.9/site-packages (from vllm==0.4.1+cu122) (9.0.0)
Requirement already satisfied: transformers>=4.40.0 in /root/anaconda3/envs/vllm/lib/python3.9/site-packages (from vllm==0.4.1+cu122) (4.40.1)
Requirement already satisfied: tokenizers>=0.19.1 in /root/anaconda3/envs/vllm/lib/python3.9/site-packages (from vllm==0.4.1+cu122) (0.19.1)
Requirement already satisfied: fastapi in /root/anaconda3/envs/vllm/lib/python3.9/site-packages (from vllm==0.4.1+cu122) (0.110.2)
Requirement already satisfied: uvicorn[standard] in /root/anaconda3/envs/vllm/lib/python3.9/site-packages (from vllm==0.4.1+cu122) (0.29.0)
Requirement already satisfied: pydantic>=2.0 in /root/anaconda3/envs/vllm/lib/python3.9/site-packages (from vllm==0.4.1+cu122) (2.7.1)
Requirement already satisfied: prometheus-client>=0.18.0 in /root/anaconda3/envs/vllm/lib/python3.9/site-packages (from vllm==0.4.1+cu122) (0.20.0)
Requirement already satisfied: tiktoken==0.6.0 in /root/anaconda3/envs/vllm/lib/python3.9/site-packages (from vllm==0.4.1+cu122) (0.6.0)
Requirement already satisfied: lm-format-enforcer==0.9.8 in /root/anaconda3/envs/vllm/lib/python3.9/site-packages (from vllm==0.4.1+cu122) (0.9.8)
Requirement already satisfied: outlines==0.0.34 in /root/anaconda3/envs/vllm/lib/python3.9/site-packages (from vllm==0.4.1+cu122) (0.0.34)
Requirement already satisfied: typing-extensions in /root/anaconda3/envs/vllm/lib/python3.9/site-packages (from vllm==0.4.1+cu122) (4.11.0)
Requirement already satisfied: filelock>=3.10.4 in /root/anaconda3/envs/vllm/lib/python3.9/site-packages (from vllm==0.4.1+cu122) (3.13.4)
Requirement already satisfied: ray>=2.9 in /root/anaconda3/envs/vllm/lib/python3.9/site-packages (from vllm==0.4.1+cu122) (2.11.0)
Collecting pynvml==11.5.0 (from vllm==0.4.1+cu122)
  Using cached pynvml-11.5.0-py3-none-any.whl.metadata (7.8 kB)
Requirement already satisfied: vllm-nccl-cu12<2.19,>=2.18 in /root/anaconda3/envs/vllm/lib/python3.9/site-packages (from vllm==0.4.1+cu122) (2.18.1.0.3.0)
Requirement already satisfied: torch==2.2.1 in /root/anaconda3/envs/vllm/lib/python3.9/site-packages (from vllm==0.4.1+cu122) (2.2.1+cu121)
Requirement already satisfied: xformers==0.0.25 in /root/anaconda3/envs/vllm/lib/python3.9/site-packages (from vllm==0.4.1+cu122) (0.0.25)
Requirement already satisfied: interegular>=0.3.2 in /root/anaconda3/envs/vllm/lib/python3.9/site-packages (from lm-format-enforcer==0.9.8->vllm==0.4.1+cu122) (0.3.3)
Requirement already satisfied: packaging in /root/anaconda3/envs/vllm/lib/python3.9/site-packages (from lm-format-enforcer==0.9.8->vllm==0.4.1+cu122) (24.0)
Requirement already satisfied: pyyaml in /root/anaconda3/envs/vllm/lib/python3.9/site-packages (from lm-format-enforcer==0.9.8->vllm==0.4.1+cu122) (6.0.1)
Requirement already satisfied: jinja2 in /root/anaconda3/envs/vllm/lib/python3.9/site-packages (from outlines==0.0.34->vllm==0.4.1+cu122) (3.1.3)
Requirement already satisfied: lark in /root/anaconda3/envs/vllm/lib/python3.9/site-packages (from outlines==0.0.34->vllm==0.4.1+cu122) (1.1.9)
Requirement already satisfied: nest-asyncio in /root/anaconda3/envs/vllm/lib/python3.9/site-packages (from outlines==0.0.34->vllm==0.4.1+cu122) (1.6.0)
Requirement already satisfied: cloudpickle in /root/anaconda3/envs/vllm/lib/python3.9/site-packages (from outlines==0.0.34->vllm==0.4.1+cu122) (3.0.0)
Requirement already satisfied: diskcache in /root/anaconda3/envs/vllm/lib/python3.9/site-packages (from outlines==0.0.34->vllm==0.4.1+cu122) (5.6.3)
Requirement already satisfied: scipy in /root/anaconda3/envs/vllm/lib/python3.9/site-packages (from outlines==0.0.34->vllm==0.4.1+cu122) (1.13.0)
Requirement already satisfied: numba in /root/anaconda3/envs/vllm/lib/python3.9/site-packages (from outlines==0.0.34->vllm==0.4.1+cu122) (0.59.1)
Requirement already satisfied: joblib in /root/anaconda3/envs/vllm/lib/python3.9/site-packages (from outlines==0.0.34->vllm==0.4.1+cu122) (1.4.0)
Requirement already satisfied: referencing in /root/anaconda3/envs/vllm/lib/python3.9/site-packages (from outlines==0.0.34->vllm==0.4.1+cu122) (0.35.0)
Requirement already satisfied: jsonschema in /root/anaconda3/envs/vllm/lib/python3.9/site-packages (from outlines==0.0.34->vllm==0.4.1+cu122) (4.21.1)
Requirement already satisfied: regex>=2022.1.18 in /root/anaconda3/envs/vllm/lib/python3.9/site-packages (from tiktoken==0.6.0->vllm==0.4.1+cu122) (2024.4.16)
Requirement already satisfied: sympy in /root/anaconda3/envs/vllm/lib/python3.9/site-packages (from torch==2.2.1->vllm==0.4.1+cu122) (1.12)
Requirement already satisfied: networkx in /root/anaconda3/envs/vllm/lib/python3.9/site-packages (from torch==2.2.1->vllm==0.4.1+cu122) (3.2.1)
Requirement already satisfied: fsspec in /root/anaconda3/envs/vllm/lib/python3.9/site-packages (from torch==2.2.1->vllm==0.4.1+cu122) (2024.3.1)
Requirement already satisfied: nvidia-cuda-nvrtc-cu12==12.1.105 in /root/anaconda3/envs/vllm/lib/python3.9/site-packages (from torch==2.2.1->vllm==0.4.1+cu122) (12.1.105)
Requirement already satisfied: nvidia-cuda-runtime-cu12==12.1.105 in /root/anaconda3/envs/vllm/lib/python3.9/site-packages (from torch==2.2.1->vllm==0.4.1+cu122) (12.1.105)
Requirement already satisfied: nvidia-cuda-cupti-cu12==12.1.105 in /root/anaconda3/envs/vllm/lib/python3.9/site-packages (from torch==2.2.1->vllm==0.4.1+cu122) (12.1.105)
Requirement already satisfied: nvidia-cudnn-cu12==8.9.2.26 in /root/anaconda3/envs/vllm/lib/python3.9/site-packages (from torch==2.2.1->vllm==0.4.1+cu122) (8.9.2.26)
Requirement already satisfied: nvidia-cublas-cu12==12.1.3.1 in /root/anaconda3/envs/vllm/lib/python3.9/site-packages (from torch==2.2.1->vllm==0.4.1+cu122) (12.1.3.1)
Requirement already satisfied: nvidia-cufft-cu12==11.0.2.54 in /root/anaconda3/envs/vllm/lib/python3.9/site-packages (from torch==2.2.1->vllm==0.4.1+cu122) (11.0.2.54)
Requirement already satisfied: nvidia-curand-cu12==10.3.2.106 in /root/anaconda3/envs/vllm/lib/python3.9/site-packages (from torch==2.2.1->vllm==0.4.1+cu122) (10.3.2.106)
Requirement already satisfied: nvidia-cusolver-cu12==11.4.5.107 in /root/anaconda3/envs/vllm/lib/python3.9/site-packages (from torch==2.2.1->vllm==0.4.1+cu122) (11.4.5.107)
Requirement already satisfied: nvidia-cusparse-cu12==12.1.0.106 in /root/anaconda3/envs/vllm/lib/python3.9/site-packages (from torch==2.2.1->vllm==0.4.1+cu122) (12.1.0.106)
Requirement already satisfied: nvidia-nccl-cu12==2.19.3 in /root/anaconda3/envs/vllm/lib/python3.9/site-packages (from torch==2.2.1->vllm==0.4.1+cu122) (2.19.3)
Requirement already satisfied: nvidia-nvtx-cu12==12.1.105 in /root/anaconda3/envs/vllm/lib/python3.9/site-packages (from torch==2.2.1->vllm==0.4.1+cu122) (12.1.105)
Requirement already satisfied: triton==2.2.0 in /root/anaconda3/envs/vllm/lib/python3.9/site-packages (from torch==2.2.1->vllm==0.4.1+cu122) (2.2.0)
Requirement already satisfied: nvidia-nvjitlink-cu12 in /root/anaconda3/envs/vllm/lib/python3.9/site-packages (from nvidia-cusolver-cu12==11.4.5.107->torch==2.2.1->vllm==0.4.1+cu122) (12.4.127)
Requirement already satisfied: annotated-types>=0.4.0 in /root/anaconda3/envs/vllm/lib/python3.9/site-packages (from pydantic>=2.0->vllm==0.4.1+cu122) (0.6.0)
Requirement already satisfied: pydantic-core==2.18.2 in /root/anaconda3/envs/vllm/lib/python3.9/site-packages (from pydantic>=2.0->vllm==0.4.1+cu122) (2.18.2)
Requirement already satisfied: click>=7.0 in /root/anaconda3/envs/vllm/lib/python3.9/site-packages (from ray>=2.9->vllm==0.4.1+cu122) (8.1.7)
Requirement already satisfied: msgpack<2.0.0,>=1.0.0 in /root/anaconda3/envs/vllm/lib/python3.9/site-packages (from ray>=2.9->vllm==0.4.1+cu122) (1.0.8)
Requirement already satisfied: protobuf!=3.19.5,>=3.15.3 in /root/anaconda3/envs/vllm/lib/python3.9/site-packages (from ray>=2.9->vllm==0.4.1+cu122) (5.26.1)
Requirement already satisfied: aiosignal in /root/anaconda3/envs/vllm/lib/python3.9/site-packages (from ray>=2.9->vllm==0.4.1+cu122) (1.3.1)
Requirement already satisfied: frozenlist in /root/anaconda3/envs/vllm/lib/python3.9/site-packages (from ray>=2.9->vllm==0.4.1+cu122) (1.4.1)
Requirement already satisfied: charset-normalizer<4,>=2 in /root/anaconda3/envs/vllm/lib/python3.9/site-packages (from requests->vllm==0.4.1+cu122) (3.3.2)
Requirement already satisfied: idna<4,>=2.5 in /root/anaconda3/envs/vllm/lib/python3.9/site-packages (from requests->vllm==0.4.1+cu122) (3.7)
Requirement already satisfied: urllib3<3,>=1.21.1 in /root/anaconda3/envs/vllm/lib/python3.9/site-packages (from requests->vllm==0.4.1+cu122) (2.2.1)
Requirement already satisfied: certifi>=2017.4.17 in /root/anaconda3/envs/vllm/lib/python3.9/site-packages (from requests->vllm==0.4.1+cu122) (2024.2.2)
Requirement already satisfied: huggingface-hub<1.0,>=0.16.4 in /root/anaconda3/envs/vllm/lib/python3.9/site-packages (from tokenizers>=0.19.1->vllm==0.4.1+cu122) (0.22.2)
Requirement already satisfied: safetensors>=0.4.1 in /root/anaconda3/envs/vllm/lib/python3.9/site-packages (from transformers>=4.40.0->vllm==0.4.1+cu122) (0.4.3)
Requirement already satisfied: tqdm>=4.27 in /root/anaconda3/envs/vllm/lib/python3.9/site-packages (from transformers>=4.40.0->vllm==0.4.1+cu122) (4.66.2)
Requirement already satisfied: starlette<0.38.0,>=0.37.2 in /root/anaconda3/envs/vllm/lib/python3.9/site-packages (from fastapi->vllm==0.4.1+cu122) (0.37.2)
Requirement already satisfied: h11>=0.8 in /root/anaconda3/envs/vllm/lib/python3.9/site-packages (from uvicorn[standard]->vllm==0.4.1+cu122) (0.14.0)
Requirement already satisfied: httptools>=0.5.0 in /root/anaconda3/envs/vllm/lib/python3.9/site-packages (from uvicorn[standard]->vllm==0.4.1+cu122) (0.6.1)
Requirement already satisfied: python-dotenv>=0.13 in /root/anaconda3/envs/vllm/lib/python3.9/site-packages (from uvicorn[standard]->vllm==0.4.1+cu122) (1.0.1)
Requirement already satisfied: uvloop!=0.15.0,!=0.15.1,>=0.14.0 in /root/anaconda3/envs/vllm/lib/python3.9/site-packages (from uvicorn[standard]->vllm==0.4.1+cu122) (0.19.0)
Requirement already satisfied: watchfiles>=0.13 in /root/anaconda3/envs/vllm/lib/python3.9/site-packages (from uvicorn[standard]->vllm==0.4.1+cu122) (0.21.0)
Requirement already satisfied: websockets>=10.4 in /root/anaconda3/envs/vllm/lib/python3.9/site-packages (from uvicorn[standard]->vllm==0.4.1+cu122) (12.0)
Requirement already satisfied: anyio<5,>=3.4.0 in /root/anaconda3/envs/vllm/lib/python3.9/site-packages (from starlette<0.38.0,>=0.37.2->fastapi->vllm==0.4.1+cu122) (4.3.0)
Requirement already satisfied: MarkupSafe>=2.0 in /root/anaconda3/envs/vllm/lib/python3.9/site-packages (from jinja2->outlines==0.0.34->vllm==0.4.1+cu122) (2.1.5)
Requirement already satisfied: attrs>=22.2.0 in /root/anaconda3/envs/vllm/lib/python3.9/site-packages (from jsonschema->outlines==0.0.34->vllm==0.4.1+cu122) (23.2.0)
Requirement already satisfied: jsonschema-specifications>=2023.03.6 in /root/anaconda3/envs/vllm/lib/python3.9/site-packages (from jsonschema->outlines==0.0.34->vllm==0.4.1+cu122) (2023.12.1)
Requirement already satisfied: rpds-py>=0.7.1 in /root/anaconda3/envs/vllm/lib/python3.9/site-packages (from jsonschema->outlines==0.0.34->vllm==0.4.1+cu122) (0.18.0)
Requirement already satisfied: llvmlite<0.43,>=0.42.0dev0 in /root/anaconda3/envs/vllm/lib/python3.9/site-packages (from numba->outlines==0.0.34->vllm==0.4.1+cu122) (0.42.0)
Requirement already satisfied: mpmath>=0.19 in /root/anaconda3/envs/vllm/lib/python3.9/site-packages (from sympy->torch==2.2.1->vllm==0.4.1+cu122) (1.3.0)
Requirement already satisfied: sniffio>=1.1 in /root/anaconda3/envs/vllm/lib/python3.9/site-packages (from anyio<5,>=3.4.0->starlette<0.38.0,>=0.37.2->fastapi->vllm==0.4.1+cu122) (1.3.1)
Requirement already satisfied: exceptiongroup>=1.0.2 in /root/anaconda3/envs/vllm/lib/python3.9/site-packages (from anyio<5,>=3.4.0->starlette<0.38.0,>=0.37.2->fastapi->vllm==0.4.1+cu122) (1.2.1)
Using cached pynvml-11.5.0-py3-none-any.whl (53 kB)
Building wheels for collected packages: vllm
  Building editable for vllm (pyproject.toml) ... error
  error: subprocess-exited-with-error

  × Building editable for vllm (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> [146 lines of output]
      /tmp/pip-build-env-rtmc4i0a/overlay/lib/python3.9/site-packages/torch/nn/modules/transformer.py:20: UserWarning: Failed to initialize NumPy: No module named 'numpy' (Triggered internally at ../torch/csrc/utils/tensor_numpy.cpp:84.)
        device: torch.device = torch.device(torch._C._get_default_device()),  # torch.device('cpu'),
      running editable_wheel
      creating /tmp/pip-wheel-mfjr6g3r/.tmp-_tv3vcke/vllm.egg-info
      writing /tmp/pip-wheel-mfjr6g3r/.tmp-_tv3vcke/vllm.egg-info/PKG-INFO
      writing dependency_links to /tmp/pip-wheel-mfjr6g3r/.tmp-_tv3vcke/vllm.egg-info/dependency_links.txt
      writing requirements to /tmp/pip-wheel-mfjr6g3r/.tmp-_tv3vcke/vllm.egg-info/requires.txt
      writing top-level names to /tmp/pip-wheel-mfjr6g3r/.tmp-_tv3vcke/vllm.egg-info/top_level.txt
      writing manifest file '/tmp/pip-wheel-mfjr6g3r/.tmp-_tv3vcke/vllm.egg-info/SOURCES.txt'
      reading manifest file '/tmp/pip-wheel-mfjr6g3r/.tmp-_tv3vcke/vllm.egg-info/SOURCES.txt'
      reading manifest template 'MANIFEST.in'
      adding license file 'LICENSE'
      writing manifest file '/tmp/pip-wheel-mfjr6g3r/.tmp-_tv3vcke/vllm.egg-info/SOURCES.txt'
      creating '/tmp/pip-wheel-mfjr6g3r/.tmp-_tv3vcke/vllm-0.4.1+cu122.dist-info'
      creating /tmp/pip-wheel-mfjr6g3r/.tmp-_tv3vcke/vllm-0.4.1+cu122.dist-info/WHEEL
      running build_py
      running build_ext
      -- The CXX compiler identification is GNU 11.4.0
      -- Detecting CXX compiler ABI info
      -- Detecting CXX compiler ABI info - done
      -- Check for working CXX compiler: /usr/bin/c++ - skipped
      -- Detecting CXX compile features
      -- Detecting CXX compile features - done
      -- Build type: RelWithDebInfo
      -- Target device: cuda
      -- Found Python: /root/anaconda3/envs/vllm/bin/python (found version "3.9.19") found components: Interpreter Development.Module
      -- Found python matching: /root/anaconda3/envs/vllm/bin/python.
      -- Found CUDA: /usr/local/cuda (found version "12.2")
      -- The CUDA compiler identification is unknown
      CMake Error at /tmp/pip-build-env-rtmc4i0a/overlay/lib/python3.9/site-packages/cmake/data/share/cmake-3.29/Modules/CMakeDetermineCUDACompiler.cmake:266 (message):
        Failed to detect a default CUDA architecture.



        Compiler output:

      Call Stack (most recent call first):
        /tmp/pip-build-env-rtmc4i0a/overlay/lib/python3.9/site-packages/torch/share/cmake/Caffe2/public/cuda.cmake:47 (enable_language)
        /tmp/pip-build-env-rtmc4i0a/overlay/lib/python3.9/site-packages/torch/share/cmake/Caffe2/Caffe2Config.cmake:87 (include)
        /tmp/pip-build-env-rtmc4i0a/overlay/lib/python3.9/site-packages/torch/share/cmake/Torch/TorchConfig.cmake:68 (find_package)
        CMakeLists.txt:67 (find_package)


      -- Configuring incomplete, errors occurred!
      Traceback (most recent call last):
        File "/tmp/pip-build-env-rtmc4i0a/overlay/lib/python3.9/site-packages/setuptools/command/editable_wheel.py", line 155, in run
          self._create_wheel_file(bdist_wheel)
        File "/tmp/pip-build-env-rtmc4i0a/overlay/lib/python3.9/site-packages/setuptools/command/editable_wheel.py", line 357, in _create_wheel_file
          files, mapping = self._run_build_commands(dist_name, unpacked, lib, tmp)
        File "/tmp/pip-build-env-rtmc4i0a/overlay/lib/python3.9/site-packages/setuptools/command/editable_wheel.py", line 280, in _run_build_commands
          self._run_build_subcommands()
        File "/tmp/pip-build-env-rtmc4i0a/overlay/lib/python3.9/site-packages/setuptools/command/editable_wheel.py", line 307, in _run_build_subcommands
          self.run_command(name)
        File "/tmp/pip-build-env-rtmc4i0a/overlay/lib/python3.9/site-packages/setuptools/_distutils/cmd.py", line 316, in run_command
          self.distribution.run_command(command)
        File "/tmp/pip-build-env-rtmc4i0a/overlay/lib/python3.9/site-packages/setuptools/dist.py", line 967, in run_command
          super().run_command(command)
        File "/tmp/pip-build-env-rtmc4i0a/overlay/lib/python3.9/site-packages/setuptools/_distutils/dist.py", line 988, in run_command
          cmd_obj.run()
        File "/tmp/pip-build-env-rtmc4i0a/overlay/lib/python3.9/site-packages/setuptools/command/build_ext.py", line 91, in run
          _build_ext.run(self)
        File "/tmp/pip-build-env-rtmc4i0a/overlay/lib/python3.9/site-packages/setuptools/_distutils/command/build_ext.py", line 359, in run
          self.build_extensions()
        File "<string>", line 177, in build_extensions
        File "<string>", line 160, in configure
        File "/root/anaconda3/envs/vllm/lib/python3.9/subprocess.py", line 373, in check_call
          raise CalledProcessError(retcode, cmd)
      subprocess.CalledProcessError: Command '['cmake', '/data/workspace/MiniCPM/vllm', '-G', 'Ninja', '-DCMAKE_BUILD_TYPE=RelWithDebInfo', '-DCMAKE_LIBRARY_OUTPUT_DIRECTORY=/tmp/tmpkga9ruse.build-lib/vllm', '-DCMAKE_ARCHIVE_OUTPUT_DIRECTORY=/tmp/tmp2xu8jtld.build-temp', '-DVLLM_TARGET_DEVICE=cuda', '-DVLLM_PYTHON_EXECUTABLE=/root/anaconda3/envs/vllm/bin/python', '-DNVCC_THREADS=1', '-DCMAKE_JOB_POOL_COMPILE:STRING=compile', '-DCMAKE_JOB_POOLS:STRING=compile=32']' returned non-zero exit status 1.
      /tmp/pip-build-env-rtmc4i0a/overlay/lib/python3.9/site-packages/setuptools/_distutils/dist.py:988: _DebuggingTips: Problem in editable installation.
      !!

              ********************************************************************************
              An error happened while installing `vllm` in editable mode.

              The following steps are recommended to help debug this problem:

              - Try to install the project normally, without using the editable mode.
                Does the error still persist?
                (If it does, try fixing the problem before attempting the editable mode).
              - If you are using binary extensions, make sure you have all OS-level
                dependencies installed (e.g. compilers, toolchains, binary libraries, ...).
              - Try the latest version of setuptools (maybe the error was already fixed).
              - If you (or your project dependencies) are using any setuptools extension
                or customization, make sure they support the editable mode.

              After following the steps above, if the problem still persists and
              you think this is related to how setuptools handles editable installations,
              please submit a reproducible example
              (see https://stackoverflow.com/help/minimal-reproducible-example) to:

                  https://github.com/pypa/setuptools/issues

              See https://setuptools.pypa.io/en/latest/userguide/development_mode.html for details.
              ********************************************************************************

      !!
        cmd_obj.run()
      Traceback (most recent call last):
        File "/root/anaconda3/envs/vllm/lib/python3.9/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 353, in <module>
          main()
        File "/root/anaconda3/envs/vllm/lib/python3.9/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 335, in main
          json_out['return_val'] = hook(**hook_input['kwargs'])
        File "/root/anaconda3/envs/vllm/lib/python3.9/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 273, in build_editable
          return hook(wheel_directory, config_settings, metadata_directory)
        File "/tmp/pip-build-env-rtmc4i0a/overlay/lib/python3.9/site-packages/setuptools/build_meta.py", line 443, in build_editable
          return self._build_with_temp_dir(
        File "/tmp/pip-build-env-rtmc4i0a/overlay/lib/python3.9/site-packages/setuptools/build_meta.py", line 395, in _build_with_temp_dir
          self.run_setup()
        File "/tmp/pip-build-env-rtmc4i0a/overlay/lib/python3.9/site-packages/setuptools/build_meta.py", line 311, in run_setup
          exec(code, locals())
        File "<string>", line 382, in <module>
        File "/tmp/pip-build-env-rtmc4i0a/overlay/lib/python3.9/site-packages/setuptools/__init__.py", line 104, in setup
          return distutils.core.setup(**attrs)
        File "/tmp/pip-build-env-rtmc4i0a/overlay/lib/python3.9/site-packages/setuptools/_distutils/core.py", line 184, in setup
          return run_commands(dist)
        File "/tmp/pip-build-env-rtmc4i0a/overlay/lib/python3.9/site-packages/setuptools/_distutils/core.py", line 200, in run_commands
          dist.run_commands()
        File "/tmp/pip-build-env-rtmc4i0a/overlay/lib/python3.9/site-packages/setuptools/_distutils/dist.py", line 969, in run_commands
          self.run_command(cmd)
        File "/tmp/pip-build-env-rtmc4i0a/overlay/lib/python3.9/site-packages/setuptools/dist.py", line 967, in run_command
          super().run_command(command)
        File "/tmp/pip-build-env-rtmc4i0a/overlay/lib/python3.9/site-packages/setuptools/_distutils/dist.py", line 988, in run_command
          cmd_obj.run()
        File "/tmp/pip-build-env-rtmc4i0a/overlay/lib/python3.9/site-packages/setuptools/command/editable_wheel.py", line 155, in run
          self._create_wheel_file(bdist_wheel)
        File "/tmp/pip-build-env-rtmc4i0a/overlay/lib/python3.9/site-packages/setuptools/command/editable_wheel.py", line 357, in _create_wheel_file
          files, mapping = self._run_build_commands(dist_name, unpacked, lib, tmp)
        File "/tmp/pip-build-env-rtmc4i0a/overlay/lib/python3.9/site-packages/setuptools/command/editable_wheel.py", line 280, in _run_build_commands
          self._run_build_subcommands()
        File "/tmp/pip-build-env-rtmc4i0a/overlay/lib/python3.9/site-packages/setuptools/command/editable_wheel.py", line 307, in _run_build_subcommands
          self.run_command(name)
        File "/tmp/pip-build-env-rtmc4i0a/overlay/lib/python3.9/site-packages/setuptools/_distutils/cmd.py", line 316, in run_command
          self.distribution.run_command(command)
        File "/tmp/pip-build-env-rtmc4i0a/overlay/lib/python3.9/site-packages/setuptools/dist.py", line 967, in run_command
          super().run_command(command)
        File "/tmp/pip-build-env-rtmc4i0a/overlay/lib/python3.9/site-packages/setuptools/_distutils/dist.py", line 988, in run_command
          cmd_obj.run()
        File "/tmp/pip-build-env-rtmc4i0a/overlay/lib/python3.9/site-packages/setuptools/command/build_ext.py", line 91, in run
          _build_ext.run(self)
        File "/tmp/pip-build-env-rtmc4i0a/overlay/lib/python3.9/site-packages/setuptools/_distutils/command/build_ext.py", line 359, in run
          self.build_extensions()
        File "<string>", line 177, in build_extensions
        File "<string>", line 160, in configure
        File "/root/anaconda3/envs/vllm/lib/python3.9/subprocess.py", line 373, in check_call
          raise CalledProcessError(retcode, cmd)
      subprocess.CalledProcessError: Command '['cmake', '/data/workspace/MiniCPM/vllm', '-G', 'Ninja', '-DCMAKE_BUILD_TYPE=RelWithDebInfo', '-DCMAKE_LIBRARY_OUTPUT_DIRECTORY=/tmp/tmpkga9ruse.build-lib/vllm', '-DCMAKE_ARCHIVE_OUTPUT_DIRECTORY=/tmp/tmp2xu8jtld.build-temp', '-DVLLM_TARGET_DEVICE=cuda', '-DVLLM_PYTHON_EXECUTABLE=/root/anaconda3/envs/vllm/bin/python', '-DNVCC_THREADS=1', '-DCMAKE_JOB_POOL_COMPILE:STRING=compile', '-DCMAKE_JOB_POOLS:STRING=compile=32']' returned non-zero exit status 1.
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building editable for vllm
Failed to build vllm
ERROR: Could not build wheels for vllm, which is required to install pyproject.toml-based projects

看日志像是 cuda 的问题，但是我这边安装安装好的，并且 WebUI Demo 能正常运行

viewhang avatar Apr 25 '24 01:04 viewhang

解决了：

export CUDA_HOME=/usr/local/cuda-12.2
export PATH="${CUDA_HOME}/bin:$PATH"

viewhang avatar Apr 25 '24 02:04 viewhang

已支持vLLM 推理，欢迎试用～ 请问哪里能找到api部署和调用示例，我尝试用vllm.entrypoints.api_server可以启动服务，但是这个支持多模态调用么

ruifengma avatar Apr 28 '24 08:04 ruifengma

不使用vllm可以吗？有没有其他openai接口的启动方式？

ye7love7 avatar May 09 '24 03:05 ye7love7

我按照install -e .的方法安装，也发生了 raise CalledProcessError(retcode, cmd) subprocess.CalledProcessError: Command '['cmake', '--build', '.', '--target', '_C', '-j', '56']' returned non-zero exit status 1. [end of output]

note: This error originates from a subprocess, and is likely not a problem with pip. 添加环境变量也不起作用。

ye7love7 avatar May 09 '24 13:05 ye7love7

我按照install -e .的方法安装，也发生了 raise CalledProcessError(retcode, cmd) subprocess.CalledProcessError: Command '['cmake', '--build', '.', '--target', '_C', '-j', '56']' returned non-zero exit status 1. [end of output]

note: This error originates from a subprocess, and is likely not a problem with pip. 添加环境变量也不起作用。

可以尝试在docker中编译

insogao avatar May 20 '24 09:05 insogao

这个api貌似没提供出来啊？咋就complete了？

ruifengma avatar May 22 '24 05:05 ruifengma

vllm 现在貌似并不支持

from openai import OpenAI

client = OpenAI()

response = client.chat.completions.create(
  model="gpt-4o",
  messages=[
    {
      "role": "user",
      "content": [
        {"type": "text", "text": "What’s in this image?"},
        {
          "type": "image_url",
          "image_url": {
            "url": "https://upload.wikimedia.org/wikipedia/commons/thumb/d/dd/Gfp-wisconsin-madison-the-nature-boardwalk.jpg/2560px-Gfp-wisconsin-madison-the-nature-boardwalk.jpg",
          },
        },
      ],
    }
  ],
  max_tokens=300,
)

print(response.choices[0])
``` 形式的接口

viewhang avatar May 22 '24 06:05 viewhang

请问支持mac系统的api 调用吗？

seasoncool avatar May 22 '24 11:05 seasoncool

add this to future plan

Cuiunbo avatar May 23 '24 11:05 Cuiunbo

I created a simple FastAPI Server which is compatible with OpenAI API. Check this and welcome any comment.

derekhsu avatar Jul 03 '24 01:07 derekhsu