tiktoken
tiktoken copied to clipboard
make decoder and sorted_token_bytes re-use existing memory
trafficstars
uses unsafe + std::mem::transmute to re-use encoder keys as decoder values and in the sorted_token_bytes list
this should be safe because all these objects share a lifetime inside CoreBPE
results in memory savings, and performance improvements in some scenarios