sensorium
sensorium copied to clipboard
Improving model-training notebook and 'trainers.py'
- The standard trainer will not stop and will not give any warning if the loss when to 'nan', it will just keep training.
- It would be nice to add in the example notebooks
'track_training': True', 'verbose': True
for thetrainer_config
(as its crucially needed for other models experiments)
Hi - yes I agree that (1) is an important feature. Has the validation accuracy been a NaN value?
Same for (2), it'd be a great addition to make it more user-friendly. If you already developed your own tracking, feel free to issue a PR against the current main.
Thanks a lot again for your helpful suggestions.