tiktoken
tiktoken copied to clipboard
Fix missing line numbers in stack traces
trafficstars
Should be a matter of putting debug = true in Cargo.toml.
before:
num_threads: 1, num_bytes: 100005605
tiktoken 6603873.890819601 bytes / s
huggingface 1668452.742767104 bytes / s
webtext encode tiktoken 13205 ms
webtext encode data_gym 31538 ms
webtext encode huggingface 52667 ms
base64 encode tiktoken 146 ms
base64 encode data_gym 1004 ms
base64 encode huggingface 470 ms
after:
num_threads: 1, num_bytes: 100005605
tiktoken 6605162.737709517 bytes / s
huggingface 1676763.3459086807 bytes / s
webtext encode tiktoken 13016 ms
webtext encode data_gym 31160 ms
webtext encode huggingface 52656 ms
base64 encode tiktoken 143 ms
base64 encode data_gym 971 ms
base64 encode huggingface 470 ms