tiktoken
tiktoken copied to clipboard
Option to use env variable for BPE_FILE location
trafficstars
Currently the BPE_FILE is hardcoded to the the path https://openaipublic.blob.core.windows.net/.... This host path is presents challenge when running in container in AWS VPC environment. It will be great if we can just specify the location of the file using ENV variable so we avoid such issues.