exo icon indicating copy to clipboard operation
exo copied to clipboard

Support local and custom models

Open leo4life2 opened this issue 1 year ago • 1 comments

Issue https://github.com/exo-explore/exo/issues/165

Done:

  • Interface changes for adding a model with HuggingFace model name
    • UI logic: put HuggingFace model name in "add model" text area (e.g. owner/modelName), click Add, select the model name in the dropdown, and start chatting.
  • Loading & running model from HuggingFace

WIP:

  • stream tokenizer decode minor bug
  • model name persisting (will probably use localSession)
  • support local models

leo4life2 avatar Sep 30 '24 05:09 leo4life2

This looks great.

What we'd really need here to get this merged is automatically sharding the model. The PyTorch implementation here does this automatically, you can see how they do it https://github.com/exo-explore/exo/pull/139 That way any language model can be supported without having to explicitly write a sharded model implementation.

AlexCheema avatar Sep 30 '24 12:09 AlexCheema