Ezzaldeen Mousa

Batch encoding decoding

2

Batch encoding and decoding: ```python from minbpe import BasicTokenizer tokenizer = BasicTokenizer() tokenizer.train(very_long_training_string, vocab_size=4096) tokenizer.encode_batch(["hello world", "bye world"]) # list[string] -> list[tokens] tokenizer.decode_batch([[1000, 2000, 3000], [1000, 2000, 3000]]) # list[tokens]...