djl-serving icon indicating copy to clipboard operation
djl-serving copied to clipboard

[AOT] Tokenizer should be optional for AOT

Open sindhuvahinis opened this issue 1 year ago • 0 comments

Description

Right now, in our default handlers, tokenizer is expected.

Right behaviour:

  1. If HF model_id is provided, then we get the tokenizer from there and just download it for them to make it ready for inference. [existing flow should cover this]
  2. If they use their own files, if tokenizer is available then we copy it, otherwise we should not expect it. We could get the tokenizer file path from the command line argument and copy it for them as well.

cc @frankfliu

sindhuvahinis avatar Jun 13 '23 20:06 sindhuvahinis