Nicolas Patry

We need a lot more information to understand why that is. Python will use the source code(and therefore a rust compiler) whenever it fails to resolve the binary. We support...

The cache key is computed by hashing the entire input (including parameters so including the regex). This is unlikely to be a cache issue. The invalid answer might have been...

Calling in @drbh on this. I know it can happen, I didn't expect 6 iteration would be enough to trigger.

TGI will always use all the allowed memory for KV-cache, to allow MANY users on the same machine. Specifying MAX_BATCH_SIZE is not used on Nvidia targets as mentionned in the...

I haven't figure out distribution correctly for Wayland vs X11 which kinds of annoys me to release at this point (since it's not just a breaking change, it's also kind...

Are you using the `sync` or `tokio` variant ? PROXY respect the HTTP_PROXY environment variable (and your system defaults). Same for the TLS. But it could be that the set...

`from_env()` builders will use the simplest version of environment variables. That's HF_ENDPOINT and HF_HOME. You can also change the endpoint manually: https://docs.rs/hf-hub/latest/hf_hub/api/sync/struct.ApiBuilder.html#method.with_endpoint

We're not entirely sure this is really the way to go. Typical deployments have multiple replicas. With CPU/disk kv-cache you need to use sticky sessions if you don't want to...

I don't think this is the solution in anyway. Adding a dependency for something so simple is not necessary and should really be avoided. Can you reproduce the behavior in...

Hi @Wehrdo , I understand the issue and sympathize with the hard to debug issue. However, I'm under the impression that the bug lies in the DataLoader, not in safetensors....