Martin Iglesias Goyanes

HOW?

7

I am relatively new to Vim, I have been using it for 4 months. I have a bunch of plugins enabled. However I can NOT make this airline statusline to...

Out of Memory Errors When Running text-generation-benchmark Despite Compliant Batch Token Limit

8

# Environment Runtime environment: - Target: x86_64-unknown-linux-gnu - Cargo version: 1.75.0 - Commit sha: c38a7d7ddd9c612e368adec1ef94583be602fc7e - Docker label: sha-6c4496a Kubernetes Cluster deployment 1 A100 GPU with 80GB RAM 12 CPU...

How to share memory among 2 GPUS for distributed inference?

10

# Environment Setup Runtime environment: Target: x86_64-unknown-linux-gnu Cargo version: 1.75.0 Commit sha: https://github.com/huggingface/text-generation-inference/commit/c38a7d7ddd9c612e368adec1ef94583be602fc7e Docker label: sha-6c4496a Kubernetes Cluster deployment 2 A100 GPU with 80GB RAM 12 CPU with 32 GB...

Feature Request: Support for LLaMa 3.1 built-in tools

3

LLaMa 3.1 can use what is called built-in tools. These built-in tools are signaled to the executor by the special token ``, also they are not user-defined JSON tools. On...

Missing code for Reliable Confidence Intervals for Information Retrieval Evaluation Using Generative A.I

2

I see the folder `high_confidence_ir_eval_using_genai` only has a `README.md` and it is missing the code referred to in the paper: https://arxiv.org/abs/2407.02464 Could you please upload the code of the experimental...

We are running LLaMa 3.1 70B on 2 A100 GPUs with 80GB of RAM each. From the logs we see that warmup phase succeeded finding the right `max_batch_total_tokens` and that...