FasterTransformer icon indicating copy to clipboard operation
FasterTransformer copied to clipboard

Serve Deberta using FasterTransformer in Triton

Open sfc-gh-zhwang opened this issue 1 year ago • 1 comments

Hi, Is there any tutorial that we can refer to so that we could serve a deberta model using fastertransformer in Triton? I think the steps would be:

  1. Convert a deberta-v2 model into fastertransformer;
  2. Dump the weights;
  3. Load it in the triton inference server;

However, I only see the step 1 with a tensorflow example.

sfc-gh-zhwang avatar Jun 28 '23 03:06 sfc-gh-zhwang

https://github.com/NVIDIA/FasterTransformer/pull/725

sfc-gh-zhwang avatar Jul 19 '23 07:07 sfc-gh-zhwang