text-generation-inference icon indicating copy to clipboard operation
text-generation-inference copied to clipboard

Published 20 hours ago • verified publisher icon huggingface

Unable to use '--rope-scaling yarn' option

Open calycekr opened this issue 11 months ago • 0 comments

Bug Report

PR: Adding yarn support https://github.com/huggingface/text-generation-inference/pull/1099 We can find 'yarn' for rope_scaling type. elif rope_scaling["type"] == "yarn":

But, there is no yarn in possible values in the tgi launcher.

error: invalid value 'yarn' for '--rope-scaling <ROPE_SCALING>'
[possible values: linear, dynamic]

No yarn support in this file. https://github.com/huggingface/text-generation-inference/blob/23fba672e8156ce19cc518470a5452a9543c56b9/launcher/src/main.rs#L107-L119

#1320 #1382

System Info

Information

  • [X] Docker
  • [X] The CLI directly

Tasks

  • [ ] An officially supported command
  • [ ] My own modifications

Reproduction

Use '--rope-scaling yarn' option.

Expected behavior

Support yarn option.

calycekr avatar Mar 18 '24 04:03 calycekr