hezeli123

[Feature] multi-nodes surpport

2

### Motivation 70B models or more larger modes should be deployed by multi-nodes( A30/L40s ...). Multi-nodes surpport will be appreciate. ### Related resources _No response_ ### Additional context _No response_

server fails in Stuck when using pipeline parallel in multi-nodes

2

### System Info 2 * 4 L40s load llama2-70B, 1 model: tensorrt_llm. using image: nvcr.io/nvidia/tritonserver:23.11-trtllm-python-py3 ### Who can help? _No response_ ### Information - [X] The official example scripts -...

# opencv code： from pathlib import Path import numpy as np import cv2 import cvcuda import logging import torch from PIL import Image img = cv2.imread('./0.jpg', cv2.IMREAD_UNCHANGED) pil_img = Image.fromarray(img.astype(np.uint8))...

[Bug] Mini-InternVL-Chat-2B-V1-5 AWQ量化后推理速度比量化前慢

5

### Checklist - [X] 1. I have searched related issues but cannot get the expected help. - [X] 2. The bug has not been fixed in the latest version. -...

[Bug] 访问一段时间后服务卡死/无响应

13

### Checklist - [X] 1. I have searched related issues but cannot get the expected help. - [x] 2. The bug has not been fixed in the latest version. ###...