Chen Wu comments

Repositories
Issues
Comments

Results 2 comments of


                                            Chen Wu

[REQUEST] Model serving via deepspeed's inference module

code ```python from transformers import pipeline import transformers import deepspeed import torch import os from transformers.models.t5.modeling_t5 import T5Block import sys import torch.distributed as dist local_rank = int(os.getenv('LOCAL_RANK', '0')) world_size =...

[Bug] Long output and issues when running benchmark_serving.py on DeepSeek-V3

> > May you try `python3 -m sglang.bench_serving --backend sglang --num-prompts 1024` instead? > > Hi, sorry for the delay, nodes were busy yesterday so just got a chance to...