tiktoken icon indicating copy to clipboard operation
tiktoken copied to clipboard

Published 20 hours ago • verified publisher icon openai

feat: Add option TIKTOKEN_FORCE_CACHE

Open Kontrolix opened this issue 2 years ago • 0 comments
trafficstars

Add taking into account the env var TIKTOKEN_FORCE_CACHE. When TIKTOKEN_FORCE_CACHE is set to "1", tiktoken will read BPE files only from the local cache.

It allows us to have control over when BPE files are downloaded from the internet.

Kontrolix avatar Jul 28 '23 12:07 Kontrolix