Ben Athiwaratkun

13B Model Out of Memory with Single Node 8 A100 GPUs

13

Hi! Thanks for contribution making this repo available :) I tried to train the 13B model with micro batch size 1, model parallelism degree 8, but unable to get it...

ImageNet Training Loss Very High (Error)

21

File "./main.py", line 166, in main train(train_loader, train_loader_len, model, ema_model, ema_model, optimizer, epoch, training_lo File "./main.py", line 492, in train assert not (np.isnan(loss.data[0]) or loss.data[0] > 1e5), 'Loss explosion: {}'.format(loss.data...

# Thank you for contributing an eval! ♥️ 🚨 Please make sure your PR follows these guidelines, __failure to follow the guidelines below will result in the PR being closed...

Error for --exp=syndigits_svhn and --exp=synsigns_gtsrb

8

I got the following errors for the two experiments above. ".../domain_datasets.py", line 24, in get_config # raise ValueError('Could not find configuration file domain_datasets.cfg') These are the two lines from batch_search_exp.sh...