SharpToken
SharpToken copied to clipboard
Support for o200k_base and gpt-4o (omni) model
Notes:
- I added in the new test plans for o200k based on output from tiktoken
- I added a SetupO200k() in CompareBenchmark (and renamed Setup to SetupCL200k), but it's not flagged as the GlobalSetup, so not currently used/benchmarked. Wasn't sure if you wanted to re-benchmark with the new encoding?
Fixed some issues that I found, all tests now passing.
Thank you so much for contribution!
I added minor fixes around failed tests in macos pipeline and will merge it here: https://github.com/dmitry-brazhenko/SharpToken/pull/43
Did it here: https://github.com/dmitry-brazhenko/SharpToken/pull/43