intel-extension-for-transformers icon indicating copy to clipboard operation
intel-extension-for-transformers copied to clipboard

Published 20 hours ago • verified publisher icon intel

[NeuralChat] Support Assisted Generation on Multi-nodes

Open letonghan opened this issue 1 year ago • 0 comments

Type of Change

feature API added:

  • /v1/assist/chat
  • /v1/assist/decode
  • /v1/assist/data_transfer

Description

Support Assisted Generation on Multi-nodes. The code framework is implemented. Details will be completed by Wangyi's team. JIRA: https://jira.devtools.intel.com/browse/NLPTOOLKIU-1126

Expected Behavior & Potential Risk

The assisted generation restful api will be able to run on multi-nodes.

How has this PR been tested?

Local. Draft PR now.

Dependency Change?

None.

letonghan avatar Feb 19 '24 03:02 letonghan