SharpToken icon indicating copy to clipboard operation
SharpToken copied to clipboard

Support for o200k_base and gpt-4o (omni) model

Open winzig opened this issue 1 year ago • 2 comments

Notes:

  • I added in the new test plans for o200k based on output from tiktoken
  • I added a SetupO200k() in CompareBenchmark (and renamed Setup to SetupCL200k), but it's not flagged as the GlobalSetup, so not currently used/benchmarked. Wasn't sure if you wanted to re-benchmark with the new encoding?

winzig avatar May 15 '24 17:05 winzig

Fixed some issues that I found, all tests now passing.

winzig avatar May 15 '24 20:05 winzig

Thank you so much for contribution!

dmitry-brazhenko avatar May 16 '24 04:05 dmitry-brazhenko

I added minor fixes around failed tests in macos pipeline and will merge it here: https://github.com/dmitry-brazhenko/SharpToken/pull/43

dmitry-brazhenko avatar May 17 '24 16:05 dmitry-brazhenko

Did it here: https://github.com/dmitry-brazhenko/SharpToken/pull/43

dmitry-brazhenko avatar May 17 '24 17:05 dmitry-brazhenko