DeepSpeed
DeepSpeed copied to clipboard
microsoft
[BUG] Bug while using deepspeed with TRL with vLLM
Describe the bug
To Reproduce
export VLLM_USE_V1=0
export NCCL_DEBUG=INFO
CUDA_VISIBLE_DEVICES=0 nohup trl vllm-serve --model google/gemma-3-4b-it --max-model-len 6656 --gpu-memory-utilization=0.9 --enable_prefix_caching True > outputs/vllm.txt & ```
CUDA_VISIBLE_DEVICES="1,2,3,4,5,6,7" nohup accelerate launch --config_file alpha/services/assistant/train/accelerate_config_8xh100.yaml alpha/services/assistant/train/train_standardhf.py > outputs/accelerate.txt &
The accelerate config is as follows:
compute_environment: LOCAL_MACHINE
debug: true
deepspeed_config:
gradient_accumulation_steps: 8
gradient_clipping: 1.0
offload_optimizer_device: cpu
offload_param_device: cpu
zero3_init_flag: false
zero_stage: 3
distributed_type: DEEPSPEED
downcast_bf16: 'no'
enable_cpu_affinity: false
machine_rank: 0
main_training_function: main
mixed_precision: bf16
num_machines: 1
num_processes: 7
rdzv_backend: static
same_network: true
tpu_env: []
tpu_use_cluster: false
tpu_use_sudo: false
use_cpu: false
The training script is this:
import datasets
import functools
from transformers import AutoModelForCausalLM
from transformers import AutoTokenizer
import trl
from alpha.services.assistant.train import dataset
from alpha.services.assistant.train import reward
if __name__ == "__main__":
processing_class = AutoTokenizer.from_pretrained('google/gemma-7b', padding_side="left")
model = AutoModelForCausalLM.from_pretrained('google/gemma-7b')
train_dataset = dataset.load_dataset(dataset.KnownClass(), datasets.Split.TRAIN)
eval_dataset = dataset.load_dataset(dataset.KnownClass(), datasets.Split.VALIDATION)
reward_fn: reward.SingleTurnComponentizedRewardFn = functools.partial(
reward.stg_rewards,
processing_class=processing_class,
max_completion_length=5120 + 1536,
)
config = trl.GRPOConfig(
num_train_epochs=16,
gradient_accumulation_steps=8,
per_device_train_batch_size=16,
per_device_eval_batch_size=16,
num_generations=16,
max_prompt_length=5120,
max_completion_length=1536,
use_cpu=False,
beta=0.04,
temperature=0.9,
bf16=True,
bf16_full_eval=True,
torch_empty_cache_steps=1,
eval_accumulation_steps=1
use_vllm=True,
)
trainer = trl.GRPOTrainer(
args=config,
train_dataset=train_dataset,
eval_dataset=eval_dataset,
processing_class=processing_class,
model=model,
reward_funcs=reward.total_reward_fn(reward_fn),
)
trainer.train()
Results in failure with the Stacktrace below:
The error in communication is between GPU 0 and GPU 1
Traceback (most recent call last):
File "/azureml-envs/designer-pytorch-2.3-train/lib/python3.10/site-packages/uvicorn/protocols/http/httptools_impl.py", line 409, in run_asgi
result = await app( # type: ignore[func-returns-value]
File "/azureml-envs/designer-pytorch-2.3-train/lib/python3.10/site-packages/uvicorn/middleware/proxy_headers.py", line 60, in __call__
return await self.app(scope, receive, send)
File "/azureml-envs/designer-pytorch-2.3-train/lib/python3.10/site-packages/fastapi/applications.py", line 1054, in __call__
await super().__call__(scope, receive, send)
File "/azureml-envs/designer-pytorch-2.3-train/lib/python3.10/site-packages/starlette/applications.py", line 112, in __call__
await self.middleware_stack(scope, receive, send)
File "/azureml-envs/designer-pytorch-2.3-train/lib/python3.10/site-packages/starlette/middleware/errors.py", line 187, in __call__
raise exc
File "/azureml-envs/designer-pytorch-2.3-train/lib/python3.10/site-packages/starlette/middleware/errors.py", line 165, in __call__
await self.app(scope, receive, _send)
File "/azureml-envs/designer-pytorch-2.3-train/lib/python3.10/site-packages/starlette/middleware/exceptions.py", line 62, in __call__
await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
File "/azureml-envs/designer-pytorch-2.3-train/lib/python3.10/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
raise exc
File "/azureml-envs/designer-pytorch-2.3-train/lib/python3.10/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app
await app(scope, receive, sender)
File "/azureml-envs/designer-pytorch-2.3-train/lib/python3.10/site-packages/starlette/routing.py", line 714, in __call__
await self.middleware_stack(scope, receive, send)
File "/azureml-envs/designer-pytorch-2.3-train/lib/python3.10/site-packages/starlette/routing.py", line 734, in app
await route.handle(scope, receive, send)
File "/azureml-envs/designer-pytorch-2.3-train/lib/python3.10/site-packages/starlette/routing.py", line 288, in handle
await self.app(scope, receive, send)
File "/azureml-envs/designer-pytorch-2.3-train/lib/python3.10/site-packages/starlette/routing.py", line 76, in app
await wrap_app_handling_exceptions(app, request)(scope, receive, send)
File "/azureml-envs/designer-pytorch-2.3-train/lib/python3.10/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
raise exc
File "/azureml-envs/designer-pytorch-2.3-train/lib/python3.10/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app
await app(scope, receive, sender)
File "/azureml-envs/designer-pytorch-2.3-train/lib/python3.10/site-packages/starlette/routing.py", line 74, in app
await response(scope, receive, send)
File "/azureml-envs/designer-pytorch-2.3-train/lib/python3.10/site-packages/starlette/responses.py", line 160, in __call__
await self.background()
File "/azureml-envs/designer-pytorch-2.3-train/lib/python3.10/site-packages/starlette/background.py", line 41, in __call__
await task()
File "/azureml-envs/designer-pytorch-2.3-train/lib/python3.10/site-packages/starlette/background.py", line 28, in __call__
await run_in_threadpool(self.func, *self.args, **self.kwargs)
File "/azureml-envs/designer-pytorch-2.3-train/lib/python3.10/site-packages/starlette/concurrency.py", line 37, in run_in_threadpool
return await anyio.to_thread.run_sync(func)
File "/azureml-envs/designer-pytorch-2.3-train/lib/python3.10/site-packages/anyio/to_thread.py", line 56, in run_sync
return await get_async_backend().run_sync_in_worker_thread(
File "/azureml-envs/designer-pytorch-2.3-train/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 2470, in run_sync_in_worker_thread
return await future
File "/azureml-envs/designer-pytorch-2.3-train/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 967, in run
result = context.run(func, *args)
File "/azureml-envs/designer-pytorch-2.3-train/lib/python3.10/site-packages/vllm/entrypoints/llm.py", line 496, in collective_rpc
return self.llm_engine.collective_rpc(method, timeout, args, kwargs)
File "/azureml-envs/designer-pytorch-2.3-train/lib/python3.10/site-packages/vllm/engine/llm_engine.py", line 2132, in collective_rpc
return self.model_executor.collective_rpc(method, timeout, args,
File "/azureml-envs/designer-pytorch-2.3-train/lib/python3.10/site-packages/vllm/executor/uniproc_executor.py", line 56, in collective_rpc
answer = run_method(self.driver_worker, method, args, kwargs)
File "/azureml-envs/designer-pytorch-2.3-train/lib/python3.10/site-packages/vllm/utils.py", line 2347, in run_method
return func(*args, **kwargs)
File "/azureml-envs/designer-pytorch-2.3-train/lib/python3.10/site-packages/trl/scripts/vllm_serve.py", line 103, in init_communicator
self.pynccl_comm = PyNcclCommunicator(pg, device=self.device)
File "/azureml-envs/designer-pytorch-2.3-train/lib/python3.10/site-packages/vllm/distributed/device_communicators/pynccl.py", line 99, in __init__
self.comm: ncclComm_t = self.nccl.ncclCommInitRank(
File "/azureml-envs/designer-pytorch-2.3-train/lib/python3.10/site-packages/vllm/distributed/device_communicators/pynccl_wrapper.py", line 277, in ncclCommInitRank
self.NCCL_CHECK(self._funcs["ncclCommInitRank"](ctypes.byref(comm),
File "/azureml-envs/designer-pytorch-2.3-train/lib/python3.10/site-packages/vllm/distributed/device_communicators/pynccl_wrapper.py", line 256, in NCCL_CHECK
raise RuntimeError(f"NCCL error: {error_str}")
RuntimeError: NCCL error: unhandled cuda error (run with NCCL_DEBUG=INFO for details)
Expected behavior
Training should not error. The error is isolated to deepspeed because if I run with
CUDA_VISIBLE_DEVICES="1,2,3,4,5,6,7" accelerate launch --multi-gpu --num-processes 7 alpha/services/assistant/train/train_standardhf.py
there are no issues
ds_report output
Please run ds_report to give us details about your setup.
ds_reportroot@tilak-8xh100:/# ds_report
[2025-04-17 23:28:52,247] [INFO] [real_accelerator.py:222:get_accelerator] Setting ds_accelerator to cuda (auto detect)
df: /root/.triton/autotune: No such file or directory
--------------------------------------------------
DeepSpeed C++/CUDA extension op report
--------------------------------------------------
NOTE: Ops not installed will be just-in-time (JIT) compiled at
runtime if needed. Op compatibility means that your system
meet the required dependencies to JIT install the op.
--------------------------------------------------
JIT compiled ops requires ninja
ninja .................. [OKAY]
--------------------------------------------------
op name ................ installed .. compatible
--------------------------------------------------
[WARNING] async_io requires the dev libaio .so object and headers but these were not found.
[WARNING] async_io: please install the libaio-dev package with apt
[WARNING] If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found.
async_io ............... [NO] ....... [NO]
fused_adam ............. [NO] ....... [OKAY]
cpu_adam ............... [NO] ....... [OKAY]
cpu_adagrad ............ [NO] ....... [OKAY]
cpu_lion ............... [NO] ....... [OKAY]
[WARNING] Please specify the CUTLASS repo directory as environment variable $CUTLASS_PATH
evoformer_attn ......... [NO] ....... [NO]
[WARNING] FP Quantizer is using an untested triton version (3.2.0), only 2.3.(0, 1) and 3.0.0 are known to be compatible with these kernels
fp_quantizer ........... [NO] ....... [NO]
fused_lamb ............. [NO] ....... [OKAY]
fused_lion ............. [NO] ....... [OKAY]
[WARNING] gds requires the dev libaio .so object and headers but these were not found.
[WARNING] gds: please install the libaio-dev package with apt
[WARNING] If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found.
gds .................... [NO] ....... [NO]
transformer_inference .. [NO] ....... [OKAY]
inference_core_ops ..... [NO] ....... [OKAY]
cutlass_ops ............ [NO] ....... [OKAY]
quantizer .............. [NO] ....... [OKAY]
ragged_device_ops ...... [NO] ....... [OKAY]
ragged_ops ............. [NO] ....... [OKAY]
random_ltd ............. [NO] ....... [OKAY]
[WARNING] sparse_attn requires a torch version >= 1.5 and < 2.0 but detected 2.6
[WARNING] using untested triton version (3.2.0), only 1.0.0 is known to be compatible
sparse_attn ............ [NO] ....... [NO]
spatial_inference ...... [NO] ....... [OKAY]
transformer ............ [NO] ....... [OKAY]
stochastic_transformer . [NO] ....... [OKAY]
--------------------------------------------------
DeepSpeed general environment info:
torch install path ............... ['/azureml-envs/designer-pytorch-2.3-train/lib/python3.10/site-packages/torch']
torch version .................... 2.6.0+cu124
deepspeed install path ........... ['/azureml-envs/designer-pytorch-2.3-train/lib/python3.10/site-packages/deepspeed']
deepspeed info ................... 0.16.4, unknown, unknown
torch cuda version ............... 12.4
torch hip version ................ None
nvcc version ..................... 12.2
deepspeed wheel compiled w. ...... torch 0.0, cuda 0.0
shared memory (/dev/shm) size .... 2.00 GB
Screenshots If applicable, add screenshots to help explain your problem.
System info (please complete the following information):
- OS: [e.g. Ubuntu 18.04] Ubuntu 22.04
- GPU count and types [e.g. two machines with x8 A100s each] one machine with 8x A100
- Interconnects (if applicable) [e.g., two machines connected with 100 Gbps IB] NA
- Python version 3.10
- Any other relevant info about your setup Training on Azure
Launcher context
Are you launching your experiment with the deepspeed launcher, MPI, or something else?
Launching with accelerate
Docker context Using a standard azure docker image
Additional context Add any other context about the problem here.
Hi @abeerag, I wonder if the master ports for vllm and deepspeed might conflict.
Can you try to add --main_process_port [PORT_NUMBER] to accelerate? (more details for options)
Thank you, will try this @tohtana - just to make sure, setting this should set it for both process 0 and the other 6 processes, right?
Hi @abeerag, it is an accelerate's option. You can set it with accelerate launch:
accelerate launch --main_process_ip ${HOST_IP} --main_process_port 12345
@abeerag, can you please update if the suggestion worked for you? Thanks!
deepspeed/ops/transformer/inference/triton/matmul_ext.py -> df: /root/.triton/autotune: No such file or directory
