tiktoken icon indicating copy to clipboard operation
tiktoken copied to clipboard

Fix missing line numbers in stack traces

Open christopherhesse opened this issue 2 years ago • 0 comments
trafficstars

Should be a matter of putting debug = true in Cargo.toml.

before:

num_threads: 1, num_bytes: 100005605
tiktoken        6603873.890819601 bytes / s
huggingface     1668452.742767104 bytes / s
webtext encode  tiktoken        13205 ms
webtext encode  data_gym        31538 ms
webtext encode  huggingface     52667 ms
base64 encode   tiktoken        146 ms
base64 encode   data_gym        1004 ms
base64 encode   huggingface     470 ms

after:

num_threads: 1, num_bytes: 100005605
tiktoken        6605162.737709517 bytes / s
huggingface     1676763.3459086807 bytes / s
webtext encode  tiktoken        13016 ms
webtext encode  data_gym        31160 ms
webtext encode  huggingface     52656 ms
base64 encode   tiktoken        143 ms
base64 encode   data_gym        971 ms
base64 encode   huggingface     470 ms

christopherhesse avatar Apr 13 '23 23:04 christopherhesse