ml-commons icon indicating copy to clipboard operation
ml-commons copied to clipboard

Published 20 hours ago • verified publisher icon opensearch-project

[Enhancement] improve the pretrained model auto truncate to max size

Open ylwu-amzn opened this issue 1 year ago • 0 comments

Some pretrained model are not supporting max truncation size. For example this model huggingface/sentence-transformers/all-MiniLM-L6-v2 can support 512 tokens, but we only support 128 in pretrained model.

ylwu-amzn avatar Dec 20 '23 06:12 ylwu-amzn