Guenter Bartsch

implemented this approach, works great for my model so far: ``` #!/usr/bin/env python # -*- coding: utf-8 -*- from pathlib import Path from lm import inference import numpy as np...

hey, cool - thanks for trying out my GPT-2 models! would be happy to hear your feedback on these. the larger GPT-2 model is still training, so if you want...

here is the command line I am using for training this model - does this help? gpt-2 de345-root data/encoded-de sp-model.model --n_embed=1024 --n_head=16 --n_layer=24 --batch_size=3 --gradient_checkpointing --save_every=5000 params.json: { "argv": "/home/bofh/projects/ai/torch/bin/gpt-2...

just a wild guess: maybe you're using a different torch version? ``` lm 0.1.0 /home/bofh/projects/ai/torch/transformer-lm pytorch-pretrained-bert 0.6.2 torch 1.2.0a0+6f6a680 ```

yes, 10.0

new release has finished uploading, available here: https://zamia.org/brain/ trained for 4.5 epochs on 27GB text corpus

I think I resumed training for this model several times over the weeks and never noticed any issue. There is however still this so far unexplained loss spike that happened...

what gpu model are you using? my settings are aimed at 11/12GB models (1080ti / titan x)

humm - interesting! :o) a bit of debug output could be enlightening here - could you apply this patch and run again? ``` diff --git a/lm/inference.py b/lm/inference.py index 8768cb7..5a4b78b 100644...

not sure about the TODO (my best guess would be that a nicer error message could be an improvement there), but the real issue seems to be the model you're...