tokenizers topic

Use custom tokenizers in spacy-transformers

megagonlabs

Building applications with LLMs through composability, in Kotlin, Scala, ...

xebia-functional

This project shows how to derive the total number of training tokens from a large text dataset from 🤗 datasets with Apache Beam and Dataflow.

sayakpaul

the small distributed language model toolkit; fine-tune state-of-the-art LLMs anywhere, rapidly

Prismadic

Implementation of the LongRoPE: Extending LLM Context Window Beyond 2 Million Tokens Paper

jshuadvd