rank_llm
rank_llm copied to clipboard
[WIP] Add SGLang Support
Pull Request Checklist
Reference Issue
Please provide the reference to issue this PR is addressing (# followed by the issue number). If there is no associated issue, write "N/A".
ref: N/A
Checklist Items
Before submitting your pull request, please review these items:
- [x] Have you followed the contributing guidelines?
- [x] Have you verified that there are no existing Pull Requests for the same update/change?
- [ ] Have you updated any relevant documentation or added new tests where needed?
PR Type
What kind of change does this PR introduce?
- [ ] Bugfix
- [x] Feature
- [ ] Code style update (formatting, local variables)
- [ ] Refactoring (no functional changes, no api changes)
- [ ] Documentation content changes
- [ ] Other...
- Description:
Instructions:
- Install SGLang locally from pjyi2147:SGLang with
pip install -e .[all]
. - Install rank_llm locally with
pip install -e .[sglang]
. - Use command to run RankZephyr on SGLang
python src/rank_llm/scripts/run_rank_llm.py --model_path=castorini/rank_zephyr_7b_v1_full --top_k_candidates=100 --dataset=dl19 --retrieval_method=SPLADE++_EnsembleDistil_ONNX --prompt_mode=rank_GPT --context_size=4096 --variable_passages --window_size=20 --sglang_batched