open_lm icon indicating copy to clipboard operation
open_lm copied to clipboard
verified publisher icon mlfoundations

Adding LLama tokenizer and S3 streaming support

Open shahromil16 opened this issue 2 years ago • 0 comments

  • Adding flags for seamless S3 streaming using BytesIO
  • Adding LLamaTokenizer for using LLama based tokenizers

shahromil16 avatar Dec 20 '23 17:12 shahromil16