How to Set "vocab_size" and "block_size" for Word Embedding?

[haibao-yu]

Hi, all, thanks for your great work.

I've been trying to configure the "vocab_size" to 1025 and "block_size" to 2580 for word token embedding and word position embedding, respectively. https://github.com/karpathy/nanoGPT/blob/325be85d9be8c81b436728a420e85796c57dba7e/model.py#L127-L128

However, it appears that the "vocab_size" needs to be larger than or equal to the "block_size" to prevent encountering a CUDA error. Do you have any suggestions or insights on how to address this issue? Many thanks.

[VatsaDev]

just set them in the train config with --block_size, vocab size is tokenizer size, depends on BPE or char, not a you setting

"vocab_size" needs to be larger than or equal to the "block_size"

Not if you go about the normal way? I have done this many times and never faced that issue? (ex. vocab 70 block 256)