Daniël de Kok

icon indicating a website link https:/danieldk.eu/

icon company @huggingface icon location Groningen, The Netherlands icon location I work on machine learning and natural language processing. I love cycling 🚴‍♂️. Machine Learning Ops Engineer @huggingface.

Results 143 comments of Daniël de Kok

Responses with unusual content.

Any chance you could test TGI 3.1.1? We fixed two prefix caching edge cases that can lead to long-term corruption.

On-The-Fly Quantization for Inference appears not to be working as per documentation.

Could you check the size of the key-value cache in both cases? The memory freed up by quantization is used to increase the size of the key-value cache, so that...

On-The-Fly Quantization for Inference appears not to be working as per documentation.

That's odd, the KV-cache size is logged unconditionally at the info level during warmup. It's only added in TGI 2.4.0, so the message wouldn't be logged in 2.0.5 (though the...