nanoGPT

nanoGPT copied to clipboard

DDP wraps the real module behind model.module, so we need to account for that when calling custom model methods.

jprobichaud

Triton Error [CUDA]: invalid argument

1

vocab_size not found in data/openwebtext/meta.pkl, using GPT-2 default of 50257 Initializing a new model from scratch number of parameters: 124.34M compiling the model... (takes a ~minute) To use data.metrics please...

zscwind

The GPU utilization very low?

2

 I am using ubuntu on window with rtx 3080. python3 train.py config/train_shakespeare_char.py --compile=False --batch_size=128 even if I increase the batch size a bit, the utilization of the GPU computation...

wx1988

Can I volunteer to add tests folder to root and some unit tests for the train script? Any preference for pytest or unit test or something else? I noticed the...

mlopsnews

Not exactly sure why, presumably the compiler is doing some dark magic, but I get *slightly* better performance when using nn.GELU() instead of `new_gelu`.

vgoklani

abrahamsangha

Loss becomes nan after training for ~2000 iterations

1

Like the title said, I've been receiving "loss: nan" after training for about 2000 iterations. The only things I changed in the script was the block size (from 1024 to...

OlympixJack

I would like to use my own data to train an AI for playing the Codenames. Is here anybody interested enough who could give me a little guidance, please? **What...

damucz

Multi GPUs training is very slow

2

I used 4 GPUs on 1 node: `torchrun --standalone --proc_per_node=4 train.py --compile=False` But, the training speed is just like 1 GPU，why?

zscwind

Segmentation fault when training shakespeare char on RTX 2080 Ti with compiled model.

1

I tried to train a baby GPT quickly with the settings provided in the config with PyTorch 2.0 on RTX 2080 Ti python train.py config/train_shakespeare_char.py --compile=True I got Segmentation fault...

jamesliu