Nicolas Patry

icon company @huggingface

Results 978 comments of Nicolas Patry

Automatically map deduplicated safetensors weights to their original values

Hi @lppllppl920 Thanks for the ping. I'm not sure why it wasn't merged.

Automatically map deduplicated safetensors weights to their original values

The fix actually doesn't work: I discovered it while testing. Fix coming soon: https://github.com/huggingface/text-generation-inference/pull/762/files#diff-2111bae5f77d998a3fe39888906b3c7be122313241ed6b69b0b0baf5abb735bbL57

NTK-Aware Scaled RoPE allows LLaMA models to have extended (8k+) context size without any fine-tuning and minimal perplexity degradation.

Oh nice. And if you want to write a PR that would be awesome too. Please be mindful that tgi code doesn't do batching the same as `transformers` meaning the...

NTK-Aware Scaled RoPE allows LLaMA models to have extended (8k+) context size without any fine-tuning and minimal perplexity degradation.

Two separate things, but we'll align with that yes.

NTK-Aware Scaled RoPE allows LLaMA models to have extended (8k+) context size without any fine-tuning and minimal perplexity degradation.

@andreaskoepf Can you provide an example where the rope scaling fails ? I'm trying few dummy examples, but I'm not sure if what I'm doing is correct or not as...

NTK-Aware Scaled RoPE allows LLaMA models to have extended (8k+) context size without any fine-tuning and minimal perplexity degradation.

@andreaskoepf the PR linked should fix it.

NTK-Aware Scaled RoPE allows LLaMA models to have extended (8k+) context size without any fine-tuning and minimal perplexity degradation.

You should be able to ! I was able to get coherent results on prompts of 6k on that model. I'm still waiting on confirmation that knows expectation from that...

NTK-Aware Scaled RoPE allows LLaMA models to have extended (8k+) context size without any fine-tuning and minimal perplexity degradation.

What model is that ?

NTK-Aware Scaled RoPE allows LLaMA models to have extended (8k+) context size without any fine-tuning and minimal perplexity degradation.

I'm bad. https://github.com/huggingface/text-generation-inference/pull/743/files

NTK-Aware Scaled RoPE allows LLaMA models to have extended (8k+) context size without any fine-tuning and minimal perplexity degradation.

Should be ok after this, could you confirm ?