LSC527

dali cuda error when running NVDeepLearningExamples with MXNET_ENABLE_CUDA_GRAPHS=1

2

code from https://github.com/NVIDIA/DeepLearningExamples/tree/master/MxNet/Classification/RN50v1.5 running with env var MXNET_ENABLE_CUDA_GRAPHS=1 ``` [1,5]: _DaliBaseIterator.__init__(self, [1,5]:2022-02-24 04:23:12,251:WARNING: DALI iterator does not support resetting while epoch is not finished. Ignoring... [1,5]:2022-02-24 04:23:12,251:INFO: Starting epoch 0...

It seems that mxnet has supported cuda graph.

No speed-up in my implementation too

3

I implemented this papaer with [torch.autograd.forward_ad](https://pytorch.org/tutorials/intermediate/forward_ad_usage.html). However, fwd gradient showed no speed-up compared to fwd+bwd.

在kubeflow dashboard中创建jupyter notebook无法找到GPU

8


ImportError running detr

1

### 🐛 Describe the bug File "/workspace/ColossalAI-Examples/image/detr/models/transformer.py", line 10, in from titans.layer.attention import DeTrAttention ImportError: cannot import name 'DeTrAttention' from 'titans.layer.attention' (/opt/conda/lib/python3.8/site-packages/titans/layer/attention/__init__.py) ### Environment _No response_

Unexpected long actor_time when train_ppo_ray

9

训练配置如下： ``` --ref_num_nodes 1 --ref_num_gpus_per_node 2 --reward_num_nodes 1 --reward_num_gpus_per_node 2 --critic_num_nodes 1 --critic_num_gpus_per_node 4 --actor_num_nodes 2 --actor_num_gpus_per_node 8 --vllm_num_engines 2 --vllm_tensor_parallel_size 4 --micro_train_batch_size 4 --train_batch_size 64 --micro_rollout_batch_size 4 --rollout_batch_size 64...

About using vLLM for generation

5

I have some thoughts about using vLLM for generation. Feel free to correct me if I were wrong. 1. Batching It seems that prompts are still passing to vllm engines...

[Bug]: RuntimeError: "cat_cuda" not implemented for 'Float8_e4m3fn'

1

### Your current environment ```text PyTorch version: 2.3.0+cu121 Is debug build: False CUDA used to build PyTorch: 12.1 ROCM used to build PyTorch: N/A OS: Ubuntu 22.04.4 LTS (x86_64) GCC...

AttributeError: 'ScalingTensor' object has no attribute 'view'

3

**What's the issue, what's expected?**: Error when using ms-amp to do llm sft. ms-amp deepspeed config: "msamp": { "enabled": true, "opt_level": "O1|O2|O3", # all tried "use_te": false } **How to...

what's the difference between ea_generate and eagenerate

1

which one should we use for generate?