improve tokenization function

[VinciGit00]

look at these sources link blog post link For hugging face models from transformers import AutoTokenizer tokenizer = AutoTokenizer.from_pretrained("meta-llama/Meta-Llama-3-8B") text = "Write your text here" tokens = tokenizer.tokenize(text) num_tokens = len(tokens) print(f"Number of tokens in your text: {num_tokens}")