tiktoken icon indicating copy to clipboard operation
tiktoken copied to clipboard

make decoder and sorted_token_bytes re-use existing memory

Open tmm1 opened this issue 1 year ago • 0 comments
trafficstars

uses unsafe + std::mem::transmute to re-use encoder keys as decoder values and in the sorted_token_bytes list

this should be safe because all these objects share a lifetime inside CoreBPE

results in memory savings, and performance improvements in some scenarios

tmm1 avatar Oct 17 '24 00:10 tmm1