aibrix

aibrix copied to clipboard

Failed to build simulator app - docker engine crashed

4

### 🚀 Feature Description and Motivation I feel this process has be to improved for easy development  ### Use Case _No response_ ### Proposed Solution _No response_

Jeffwan

Adding scaling down to 0 case Gateway handling

8

### 🚀 Feature Description and Motivation The autoscaler should support scaling down to 0. When a new request arrives, we should have an activator component intercepts the request and initializes...

nwangfw

StreamLoader supports each TP process to only read the corresponding part of the model file

1

### 🚀 Feature Description and Motivation Currently, each TP (Tensor Parallism) process in StreamLoader reads all model files, resulting in duplicate file transfers and reads, which reduces the overall loading...

brosoul

StreamLoader library support device_map to allocate tensor to different device

2

### 🚀 Feature Description and Motivation In the parameter list of the loading tensor in the StreamLoader library, `device` should be refined to `device_map`. And StreamLoader library should support device_map...

brosoul

StreamLoader support multi processes and multi threads

1

### 🚀 Feature Description and Motivation The network IO of a single process may have an upper limit, and adopting a multi process and multi thread approach can better utilize...

brosoul

Speedup downloading models by parallel downloading of multiple files

1

### 🚀 Feature Description and Motivation AI Runtime should support download models by parallel downloading of multiple files ### Use Case _No response_ ### Proposed Solution _No response_

brosoul

[RFC] Model management in Runtime

4

### Summary The current runtime provides the ability to download model files from different sources, but lacks management capabilities for model files. ### Motivation This issue aims to provide model...

brosoul

Support lora metrics in vLLM

2

### 🚀 Feature Description and Motivation FYI ### Use Case _No response_ ### Proposed Solution _No response_

Jeffwan

StreamLoader can persist models to disk in bypass

1

### 🚀 Feature Description and Motivation StreamLoader can persist models to disk in bypass. This way can avoid the network transmission overhead of loading the model from the current machine...

brosoul

Autoscaler doesn't update the CR metrics change in memory correspondingly

1

### 🐛 Describe the bug  reproduce steps 1. define one metric name 2. update metric metric name and apply the cr 3. monitor the autoscaler logs ### Steps to...

Jeffwan