Cal Mitchell

Is there a way to choose an SSH key pair?

1

I'd like to be able to SSH into the machine manually to verify things. Is there a way to specify which SSH pair the EC2 instance uses? Sorry if this...

PyTorch nn.LayerNorm now takes bias arg - removed custom class

1

Hi, I noticed that the PyTorch `nn.LayerNorm` class now takes a `bias` arg. This PR removes the custom `LayerNorm` class and replaces it with the built-in. I tested the qualitative...

Would like to contribute FSDP functionality

2

Hi @karpathy, I recently modified this repo to shard everything with PyTorch FSDP and would like to contribute this functionality to the project. If I cleaned up the code and...

Seeking guidance on continuing pretraining

4

Hello, thanks very much for the excellent work on this repo. There are several examples showing how to create a question-response style dataset, but I can't immediately tell how to...

feat: added packed continued pretraining dataset functionality

5

#### Context This PR seeks to resolve issue #809 by adding the ability to pack and tokenize input/label pairs from a Huggingface dataset and then continue a model's pretraining phase...

How to save a trained model so it can be loaded with HF `from_pretrained()`?

29

I'm finding this repo to be a user friendly, extensible, memory efficient solution for training/fine-tuning models. However, when it comes to inference, there is a usability gap that could be...

Potential readme issue - falls back to original dtype, not fp32

1

In the docs, it says that when quantizing to anything other than int8, many operations will fall back to fp32. However, looking through the code (and inserting some print lines)...

[Bug]: bcp giving "An existing connection was forcibly closed by the remote host." error

8

### What happened? I am loading a large amount of tables in a for loop with bcp. It is working well most of the time but sometimes errors out (on...