minGPT
minGPT copied to clipboard
karpathy
A minimal PyTorch re-implementation of the OpenAI GPT (Generative Pretrained Transformer) training
I saw that there was not a requirements.txt file, so I created one by looking at all the .py files and gathering all non-built-in libraries used.
https://github.com/karpathy/minGPT/blob/37baab71b9abea1b76ab957409a1cc2fbfba8a26/mingpt/model.py#L42 Why do we need an additional linear transformation after the MHA and before the MLP when the dimensions are the same? (I understand that this is how the initial...
"Hi, I am reading the GPT-2 paper and encountering a problem with the following phrase related to implementation: 'A modified initialization method is used to account for the accumulation on...
https://github.com/karpathy/minGPT/blob/37baab71b9abea1b76ab957409a1cc2fbfba8a26/mingpt/model.py#L64 I accidentally changed the softmax dimension to -2 instead of -1 and got incredibly low losses on both the training and validation set when using the [tiny_shakespeare](https://www.tensorflow.org/datasets/catalog/tiny_shakespeare) dataset. However,...
codes, comments, mathematical and physical books for the development of CAX software to train miniGPT to be the AI assistant of our project, www.opendigitaltwin.top
DEMO: minGPT on tinygrad Do not merge. demonstration only. https://github.com/ziliangpeng/tinyGPT # You like karpathy? You like geohot? You love tinyGPT! ❤️ 
```python import torch from mingpt.bpe import BPETokenizer tokenizer = BPETokenizer() print(tokenizer("")) # tensor([[ 27, 91, 437, 1659, 5239, 91, 29]]) print(tokenizer.decode(torch.tensor([50256]))) # '' print(tokenizer(tokenizer.decode(torch.tensor([50256])))) # tensor([[ 27, 91, 437, 1659,...
