Megatron-DeepSpeed

Megatron-DeepSpeed copied to clipboard

Hello, what version of the megatron-lm library is your code modified?

4thGardenOfQMH

Is this assertion for mask wrong?

1

I got an `AssertionError: Mask is silently ignored due to the use of a custom kernel` when training GPT-2 with `examples/pretrain_gpt.sh`. This line leads to the assertion error: https://github.com/bigscience-workshop/Megatron-DeepSpeed/blob/8387ae17c4704f6579f88a84500b535d19d7fbbf/megatron/model/fused_softmax.py#L191 Is...

yinfangchen

The given group does not exist pytorch

2

Do you know why i got this problem with `pretrain_gpt_single_node.sh`? I'm setting `N_GPUS=1` and got ``` File "/opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py", line 191, in _get_group_rank raise RuntimeError("The given group does not exist") RuntimeError:...

germanjke

Add FlashAttention

3

This PR aims to add an option to use [FlashAttention](https://github.com/HazyResearch/flash-attention). Inspired by https://github.com/NVIDIA/Megatron-LM/pull/267 cc @thomasw21

NouamaneTazi

Hello, can Megatron-DeepSpeed pre-train llama2? Can give a sample script?

13416157913

RuntimeError: Error building extension 'scaled_upper_triang_masked_softmax_cuda'

1

zll0000

d norm: nan | actual seqlen: 2048 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1.886 | TFLOPs: 78.46 | iteration 5426/...

alphanlp

hi, I looked up a lot of information. but I still don't understand the difference between zero-3 and megatron with zero-2. they all split the model.

nicosouth

Exception: cuda rng state model-parallel-rng is not added

1

i start the job the i met this error: cuda: 12.0 torch: 1.14 deepspeed --num_gpus 2 pretrain_gpt_v2.py --tensor-model-parallel-size 1 --pipeline-model-parallel-size 1 --distributed-backend nccl --num-layers 2 --hidden-size 64 --num-attention-heads 2 --seq-length...

520jefferson

Hello. When using tensor parallel on bloom (tp_size = 8), we find that the cross_entropy loss computed by mpu.cross_entropy is different from torch.nn.functional.cross_entropy. The difference is about 1% for our...

robin087