Rob Leidle

Iterating on Paged Response objects after their "client" has been closed

3

The issue is that PagedResponseIterators cannot be used once their associated service client is closed. This appears to be a pattern in this repo so I am opening this issue....

heartbeat thread is sometimes not terminated upon client closing

10

I am running into a frequent issue where the heartbeat thread does not get interrupted properly when close my SSHClient object. This happens often when the sshClient.connect is refused (I...

Option for relaxing the constraint that the stack id for the builder, run, and build image must all match

2

### Description We have pre-existing run images, each with a different stack id, that we would like to use with a single builder. Since a builder can only have a...

Out of memory when launching server with larger values for --max-batch-prefill-tokens yet GPU memory is available

5

### System Info ``` text-generation-launcher --env 2024-04-01T20:49:45.871764Z INFO text_generation_launcher: Runtime environment: Target: x86_64-unknown-linux-gnu Cargo version: 1.75.0 Commit sha: e6bb3ff81fd670ad2f54904676f8165367dd47f8 Docker label: sha-e6bb3ff ``` ### Information - [X] Docker - [...

Error during cuda_graph_warmup(...) when increasing allowed tokens

1

### System Info ``` Target: x86_64-unknown-linux-gnu Cargo version: 1.75.0 Commit sha: 4ee0a0c4010b6e000f176977648aa1749339e8cb Docker label: sha-4ee0a0c nvidia-smi: Tue Apr 2 17:34:07 2024 +---------------------------------------------------------------------------------------+ | NVIDIA-SMI 535.161.08 Driver Version: 535.161.08 CUDA Version:...

A common problem across buildpacks is that when combining related buildpacks into a docker image, many small binaries are created, each with almost the same content. The request is for...

When the backend is `tgi`, the `build_url(...)` function is simply returning the supplied URL parameter. When a user passes in the base URL of their TGI server, the result is...

## Description db.db.serialize(...) guarantees that each statement runs in order. However, the function itself will terminate execution long before all the statements are executed. Therefore, we should only resolve the...

Currently, to enable debug level tracing, one sets the environment variable `RUST_LOG` to `debug`. This aloso increases the log level to be debug which often is not desired. The request...