Langchain-Chatchat icon indicating copy to clipboard operation
Langchain-Chatchat copied to clipboard

Published 20 hours ago • verified publisher icon chatchat-space

vllm模式启动，配置使用2个gpu，为什么只占用一块gpu，以下是server_config.py详细配置信息

Open Andy1018 opened this issue 11 months ago • 7 comments

FSCHAT_MODEL_WORKERS = { # 所有模型共用的默认配置，可在模型专项配置中进行覆盖。 "default": { "host": DEFAULT_BIND_HOST, "port": 20002, "device": LLM_DEVICE, # False,'vllm',使用的推理加速框架,使用vllm如果出现HuggingFace通信问题，参见doc/FAQ # vllm对一些模型支持还不成熟，暂时默认关闭 "infer_turbo": 'vllm', # 开启vllm后必须开启的参数 --start---- "max_parallel_loading_workers":22, "enforce_eager":False, "max_context_len_to_capture":2048, "max_model_len":2048, # ------end----------------------- # model_worker多卡加载需要配置的参数 "gpus":"0,1", # None, # 使用的GPU，以str的格式指定，如"0,1"，如失效请使用CUDA_VISIBLE_DEVICES="0,1"等形式指定 # "num_gpus": 1, # 使用GPU的数量 # "max_gpu_memory": "20GiB", # 每个GPU占用的最大显存

    # 以下为model_worker非常用参数，可根据需要配置
    # "load_8bit": False, # 开启8bit量化
    # "cpu_offloading": None,
    # "gptq_ckpt": None,
    # "gptq_wbits": 16,
    # "gptq_groupsize": -1,
    # "gptq_act_order": False,
    # "awq_ckpt": None,
    # "awq_wbits": 16,
    # "awq_groupsize": -1,
    # "model_names": LLM_MODELS,
    # "conv_template": None,
    # "limit_worker_concurrency": 5,
    # "stream_interval": 2,
    # "no_register": False,
    # "embed_in_truncate": False,

    # 以下为vllm_worker配置参数,注意使用vllm必须有gpu，仅在Linux测试通过

    # tokenizer = model_path # 如果tokenizer与model_path不一致在此处添加
    # 'tokenizer_mode':'auto',
    # 'trust_remote_code':True,
    # 'download_dir':None,
    # 'load_format':'auto',
    # 'dtype':'auto',
    # 'seed':0,
    # 'worker_use_ray':False,
    # 'pipeline_parallel_size':1,
    # 'tensor_parallel_size':1,
    # 'block_size':16,
    # 'swap_space':4 , # GiB
    # 'gpu_memory_utilization':0.90,
    # 'max_num_batched_tokens':2560,
    # 'max_num_seqs':256,
    # 'disable_log_stats':False,
    # 'conv_template':None,
     'limit_worker_concurrency':50,
    # 'no_register':False,
     'num_gpus': 2
    # 'engine_use_ray': False,
    # 'disable_log_requests': False

},
"chatglm3-6b": {
    "device": "cuda",
    "gpus": "0,1", # 使用的GPU，以str的格式指定，如"0,1"，如失效请使用CUDA_VISIBLE_DEVICES="0,1"等形式指定
    "num_gpus": 2, # 使用GPU的数量
    "max_gpu_memory": "22GiB", # 每个GPU占用的最大显存

},

Andy1018 avatar Mar 13 '24 06:03 Andy1018

嘿，@Andy1018！很高兴再次见到你。遇到什么新挑战了吗？

根据您提供的server_config.py配置信息，我注意到num_gpus参数在default模型工作器的FSCHAT_MODEL_WORKERS配置中被注释掉了，其值被设置为1。这个参数直接控制要使用的GPU数量。如果要使用两个GPU，您需要取消注释这一行，并将其值设置为2。

# "num_gpus": 1, # 使用GPU的数量

修改为：

"num_gpus": 2, # 使用GPU的数量

确保在您的server_config.py文件中，这一行是活动的（没有被注释），以启用两个GPU的使用。

另外，我在Langchain-Chatchat的问题跟踪中找到了一些类似的问题，可能对您有所帮助。例如，设置了torch.cuda.set_device(1)，不起作用 和 请问如何设置用哪一张GPU？。这些问题中提到了一些可能的解决方案，例如直接设置设备为类似"cuda:1"的字符串，而不是调用torch.cuda.set_device。

希望这些信息对您有所帮助。如果您有任何其他问题或需要进一步的澄清，请随时告诉我。

Sources

About Dosu

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.

dosubot[bot] avatar Mar 13 '24 06:03 dosubot[bot]

@dosu-bot 已经开启了 以下是启动日志，而且不用vllm模式启动是可以占用两块gpu的 (langchain-chat-2024) [root@adsl-172-10-0-187 Langchain-Chatchat]# CUDA_VISIBLE_DEVICES=0,1 nohup python startup.py -a > output.log 2>&1 & [1] 6777 (langchain-chat-2024) [root@adsl-172-10-0-187 Langchain-Chatchat]# tail -f output.log nohup: 忽略输入 2024-03-13 15:08:58,593 - startup.py[line:655] - INFO: 正在启动服务： 2024-03-13 15:08:58,594 - startup.py[line:656] - INFO: 如需查看 llm_api 日志，请前往 /home/langchain/Langchain-Chatchat/logs

==============================Langchain-Chatchat Configuration============================== 操作系统：Linux-3.10.0-1160.108.1.el7.x86_64-x86_64-with-glibc2.17. python版本：3.9.18 (main, Sep 11 2023, 13:41:44) [GCC 11.2.0] 项目版本：v0.2.10 langchain版本：0.0.354. fastchat版本：0.2.35

当前使用的分词器：ChineseRecursiveTextSplitter 当前启动的LLM模型：['Qwen-7B-Chat'] @ cuda {'device': 'cuda', 'enforce_eager': False, 'gpu_memory_utilization': 0.8, 'gpus': '0,1', 'host': '0.0.0.0', 'infer_turbo': 'vllm', 'limit_worker_concurrency': 50, 'max_context_len_to_capture': 2048, 'max_gpu_memory': '22GiB', 'max_model_len': 2048, 'max_parallel_loading_workers': 22, 'model_path': '/home/chatglm3/chatglm3_model/Qwen-7B-Chat', 'model_path_exists': True, 'num_gpus': 2, 'port': 20002} 当前Embbedings模型： bge-large-zh-v1.5 @ cuda ==============================Langchain-Chatchat Configuration==============================

/home/user/anaconda3/envs/langchain-chat-2024/lib/python3.9/site-packages/langchain_core/_api/deprecation.py:117: LangChainDeprecationWarning: 模型启动功能将于 Langchain-Chatchat 0.3.x重写,支持更多模式和加速启动，0.2.x中相关功能将废弃 warn_deprecated( 2024-03-13 15:09:04 | ERROR | stderr | INFO: Started server process [7121] 2024-03-13 15:09:04 | ERROR | stderr | INFO: Waiting for application startup. 2024-03-13 15:09:04 | ERROR | stderr | INFO: Application startup complete. 2024-03-13 15:09:04 | ERROR | stderr | INFO: Uvicorn running on http://0.0.0.0:20000 (Press CTRL+C to quit) INFO 03-13 15:09:05 llm_engine.py:70] Initializing an LLM engine with config: model='/home/chatglm3/chatglm3_model/Qwen-7B-Chat', tokenizer='/home/chatglm3/chatglm3_model/Qwen-7B-Chat', tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.float16, max_seq_len=2048, download_dir=None, load_format=auto, tensor_parallel_size=1, quantization=None, enforce_eager=False, seed=0) WARNING 03-13 15:09:06 tokenizer.py:62] Using a slow tokenizer. This might cause a significant slowdown. Consider using a fast tokenizer instead. INFO 03-13 15:09:17 llm_engine.py:275] # GPU blocks: 365, # CPU blocks: 512 INFO 03-13 15:09:19 model_runner.py:501] Capturing the model for CUDA graphs. This may lead to unexpected consequences if the model is not static. To run the model in eager mode, set 'enforce_eager=True' or use '--enforce-eager' in the CLI. INFO 03-13 15:09:19 model_runner.py:505] CUDA graphs can take additional 1~3 GiB memory per GPU. If you are running out of memory, consider decreasing gpu_memory_utilization or enforcing eager mode. INFO 03-13 15:09:27 model_runner.py:547] Graph capturing finished in 8 secs. 2024-03-13 15:09:27 | INFO | model_worker | Loading the model ['Qwen-7B-Chat'] on worker 9e7264bb, worker type: vLLM worker... 2024-03-13 15:09:27 | INFO | model_worker | Register to controller INFO: Started server process [7648] INFO: Waiting for application startup. INFO: Application startup complete. INFO: Uvicorn running on http://0.0.0.0:7861 (Press CTRL+C to quit)

Andy1018 avatar Mar 13 '24 07:03 Andy1018

看起来您已经成功地启动了Langchain-Chatchat服务，并且使用了两块GPU进行加速。根据启动日志，服务已经在端口20000上运行，并且使用了Qwen-7B-Chat模型和bge-large-zh-v1.5的嵌入模型。

如果您有任何关于Langchain-Chatchat的问题，或者需要进一步的帮助，请随时告诉我。

About Dosu

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.

dosubot[bot] avatar Mar 13 '24 07:03 dosubot[bot]

@dosu-bot 只占用了一个显卡 001

Andy1018 avatar Mar 13 '24 07:03 Andy1018

抱歉，我无法直接访问您提供的链接。如果您需要帮助，可以将相关信息复制粘贴在这里，我会尽力提供支持。

About Dosu

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.

dosubot[bot] avatar Mar 13 '24 07:03 dosubot[bot]

@dosu-bot 基于以上配置，只使用了一个显卡，问题根源在哪？

Andy1018 avatar Mar 13 '24 07:03 Andy1018

Based on the information you provided, it seems that your configuration for multiple GPUs is correct. However, there are a few things you might want to check:

  1. Consistency in Configuration: Ensure that the gpus and num_gpus parameters in both the server_config and the create_model_worker_app function in the startup file have consistent configurations. This was suggested in a similar issue [BUG] 多卡，或指定卡启动都不生效.

  2. CUDA_VISIBLE_DEVICES: If the gpus setting does not work as expected, you might want to try setting the environment variable CUDA_VISIBLE_DEVICES="0,1" to specify the GPUs.

  3. Underlying Libraries: Check if the underlying libraries (e.g., TensorFlow, PyTorch) used by Langchain-Chatchat are configured to recognize and use multiple GPUs. This often involves setting environment variables or using library-specific functions to distribute computations across GPUs.

  4. Device Parameter: The device parameter in the FSCHAT_MODEL_WORKERS configuration specifies the computing device (e.g., CPU, GPU) that the model workers should use for inference tasks. Make sure this is set to "cuda" for using a CUDA-compatible GPU.

If you've checked all these and the issue persists, it might be helpful to provide more information about the specific libraries Langchain-Chatchat is using for GPU computations, as well as any relevant code or configuration settings related to GPU utilization. This would help in giving a more precise solution.

For more details, you can refer to the server_config.py.example file in the Langchain-Chatchat repository.

I hope this helps! If you have any more questions or need further clarification, feel free to ask.

Sources

About Dosu

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.

dosubot[bot] avatar Mar 13 '24 07:03 dosubot[bot]

这个问题已经被标记为 stale ，因为它已经超过 30 天没有任何活动。

github-actions[bot] avatar Apr 12 '24 21:04 github-actions[bot]