text-generation-inference icon indicating copy to clipboard operation
text-generation-inference copied to clipboard

Published 20 hours ago • verified publisher icon huggingface

max_total_tokens is hardcoded to 1512 and cant be changed from SageMaker.

Open monuminu opened this issue 1 year ago • 0 comments

System Info

max_total_tokens is hardcoded to 1512 and cant be changed from SageMaker.

Information

  • [ ] Docker
  • [ ] The CLI directly

Tasks

  • [ ] An officially supported command
  • [ ] My own modifications

Reproduction

If the token size exceeds 1512 it gives error.

Expected behavior

There should be a option to increase the same using env variable.

monuminu avatar Jun 06 '23 17:06 monuminu