Harry Mellor

icon company @huggingface @vllm-project icon location ML Engineer @huggingface

Results 298 comments of Harry Mellor

Add config classes to API docs

Ok, currently working on improving the left sidebar so that the entire API can be navigated properly

Improve configs - `ModelConfig`

I think I need a better solution to the hashing error. If possible, it would be better not to need `ModelConfig` to be hashable at all.

Improve configs - `ModelConfig`

Since this PR has become quite big, I've been splitting it up. You can see the description for the sub-PRs.

Default to `generation_config` from model

Good point, the latest change updates the default in: - `EngineArgs` (including the CLI arg) - `ModelConfig`

Default to `generation_config` from model

Failing tests are likely due to changes in sampling behaviour

Default to `generation_config` from model

Interestingly the "V1 Test" will timeout because `ModelConfig.get_diff_sampling_param()` is called for every request. The slow part of `ModelConfig.get_diff_sampling_param()` is `ModelConfig.try_get_generation_config()`, which reads the default config from disk using `GenerationConfig.from_pretrained`. This...

[Core] Async Scheduling X Spec Decoding Compatibility

These conflicts are caused by our migration to `ruff`. Please see https://vllm-dev.slack.com/archives/C07R5Q1Q2BB/p1759663228844749 which contains detailed instructions to make updating your branch as painless as possible.

[Core] Async Scheduling X Spec Decoding Compatibility

I'm not the right person to review the speculative decoding part of the PR, I'll leave that to the codeowners (@benchislett @luccafong)

[Doc]: provide docker-compose.yml for multi-node serving

Could you use the contents of `run_cluster.sh` as the contents of your docker service?

[fix]: Dockerfile.ppc64le fixes for opencv-python and hf-xet

I've unblocked the `ppc64le` test