huggingface-vscode-endpoint-server icon indicating copy to clipboard operation
huggingface-vscode-endpoint-server copied to clipboard

Published 20 hours ago • verified publisher icon LucienShui

Refactor generators and add ct2fast support

Open piratos opened this issue 2 years ago • 2 comments

Hello, in my fork I:

  • refactored the generators
  • Added support loading ctranslate2 based models (starcoderct2fast) which are incredibly fast on consumer hardware
  • Added support to finding the model type and returning the correct class in the main function (from local or HF hub)
  • support for websocket streaming

it is a WIP but if you want any of these features I ll be happy to create a proper PR

piratos avatar Jun 18 '23 10:06 piratos

Sorry for taking so long to reply to you. In fact, I encourage everyone to submit pull requests directly, and I will carefully review each one.

LucienShui avatar Jul 15 '23 09:07 LucienShui

Anything happened to this ticket?

thanhnew2001 avatar Nov 06 '23 06:11 thanhnew2001