minbpe

minbpe copied to clipboard

Add error handling to the `load` method in `base.py`, specifically handling cases where the model file format might be incorrect or corrupted. Modifications include: 1. Checking the model file version...

halannhile

Huggingface already has an efficient implementation of this?

3

When Karpathy claimed an efficient implementation of the BPE optimizer doesn't exist, I did some research and found this on Hugging Face: https://github.com/huggingface/tokenizers/blob/main/tokenizers/src/models/bpe/trainer.rs Isn't this exactly what Karpathy was creating?

laurislopata

Using minBPE token encoded sentence vectors need to be padded

1

Without the padding, the sentences end up being different sizes and we get stacking errors at data loading time.

elevateclub

Instead of finding the one pair with the highest frequency and merging it at each step, do the highest N pairs

2

I've been experimenting with my own little BPE implementations in other programming languages. It seems that major bottlenecks are counting the frequencies of the pairs each iteration and merging the...

hippietrail

👋🏻🙋🏻‍♂️ **Hello @karpathy ,** Firstly, I apologize for creating an "issue," but it seems to be an effective way to reach you. I have been following your lectures since last...

AayushSameerShah

Much faster Regex tokenization using c++ and ctypes

2

i implemented a much faster training/tokenization algorithm in c++ and called the functions using ctypes, so that they can be used conveniently in python. The performance gain is huge, i...

JohannesVod

`minbpe-rs`: A pure Rust implementation of `minbpe`

2

[Gregor Purdy (@gnp)](https://github.com/gnp) is working on a Rust version of `minbpe`: [minbpe-rs](https://github.com/gnp/minbpe-rs) The Rust crate (similar to a package in Python) contains the three tokenizers currently included in the Python...

shubham0204

[Test workflow] Add test workflow

2

A little more reliability I guess?

sayakpaul

It appears that the decode() method in the GPT4Tokenizer class does not handle special tokens. I submitted a pull request (#63) with some updated code, but also wanted to post...

Vakarva

…okens. Copy-pasted decode() method from class RegexTokenizer to allow handling of special tokens; and added two lines to "unshuffle" bytes objects.

Vakarva