ztang2370

[Feature] microbatch tokenization

3

## Essential Elements of an Effective PR Description Checklist - [x] The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)". - [x]...

[Feat] [WIP] Ollama integration

1

Issue https://github.com/ovg-project/kvcached/issues/81 - Adds initial support for integrating Ollama with kvcached. - Verified workflow locally on a single CUDA GPU (RTX 3090). - Current implementation runs end-to-end but requires: -...

[Example] Add vllm semantic router example

6

Issue: https://github.com/ovg-project/kvcached/issues/91 This is WIP, a basic working example. TODO: 1. Include the sleep and wakeup functionality based on the traffic monitoring status. 2. As vllm semantic router doesn't have...