lkchen

- [ ] My submission is formatted according to the guidelines in the [contributing guide](/CONTRIBUTING.md) - [ ] My addition is ordered alphabetically - [ ] My submission has a...

[V1][Molmo] Fix get_multimodal_embeddings() in molmo.py

2

Expected: get_multimodal_embeddings() should return list[Tensor] for `GPUModelRunner` to iterate. Actual: prior to this PR, molmo's _get_mm_embeds() returns a list thus get_multimodal_embeddings() returns a list of list. The issue is surfaced...

[benchmark][structured output] Add offline benchmark script for structured output

1

* Reuse some flags from serving version * Tested that serving benchmark still work in V0 - V1 serving benchmark is broken, but it was already broken on main branch...

[P/D] Asynchronously do _nixl_handshake

1

## Essential Elements of an Effective PR Description Checklist - [x] The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)". - [x]...

WIP [P/D] Use ThreadPoolExecutor to do handshake for each P-D pair

2

## Essential Elements of an Effective PR Description Checklist - [x] The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)". - [x]...

[Misc] Add type alias `ReqId` and `EngineId` for better readability

2

## Essential Elements of an Effective PR Description Checklist - [x] The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)". - [...

[BugFix][P/D] Fix for cases where _recving_transfers can be cleaned up when *all* transfer done

3

## Essential Elements of an Effective PR Description Checklist - [x] The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)". - [x]...