gpt_bpe icon indicating copy to clipboard operation
gpt_bpe copied to clipboard
verified publisher icon wbrown

Update transpiler and allow creation of gobs

Open Rexwang8 opened this issue 1 year ago • 0 comments

Summary

gpt_bpe is intended to be transpiled into JavaScript via GopherJS to be used with goose_web. This PR adds function to allow serialization to and from gobs, which are faster than json or other native JS formats.

Sister PR found here: https://github.com/gooseai/goose-web/pull/139

Changes

  1. Added functions for the serialization and de-serialization of encoders to gobs.
  2. Added tests testing # 1
  3. Removed the go-humanize package in favor of a locally written function that doesn't use math.pow as it transpiles strangely in GopherJS
  4. Added a makefile instruction for building GopherJS and updated readme with a short related blurb.
  5. Added gobs for 5 encoders into the resources/embedded folder and embedded said gobs into gpt_bpe.js
  6. Update js.go to work with the current iteration of goose-web and provide some utilties to check the load state of encoders.

Rexwang8 avatar Sep 24 '24 15:09 Rexwang8