rank_llm icon indicating copy to clipboard operation
rank_llm copied to clipboard

[WIP] Add SGLang Support

Open pjyi2147 opened this issue 4 months ago • 1 comments

Pull Request Checklist

Reference Issue

Please provide the reference to issue this PR is addressing (# followed by the issue number). If there is no associated issue, write "N/A".

ref: N/A

Checklist Items

Before submitting your pull request, please review these items:

  • [x] Have you followed the contributing guidelines?
  • [x] Have you verified that there are no existing Pull Requests for the same update/change?
  • [ ] Have you updated any relevant documentation or added new tests where needed?

PR Type

What kind of change does this PR introduce?

  • [ ] Bugfix
  • [x] Feature
  • [ ] Code style update (formatting, local variables)
  • [ ] Refactoring (no functional changes, no api changes)
  • [ ] Documentation content changes
  • [ ] Other...
    • Description:

Instructions:

  1. Install SGLang locally from pjyi2147:SGLang with pip install -e .[all].
  2. Install rank_llm locally with pip install -e .[sglang].
  3. Use command to run RankZephyr on SGLang
python src/rank_llm/scripts/run_rank_llm.py  --model_path=castorini/rank_zephyr_7b_v1_full --top_k_candidates=100 --dataset=dl19 --retrieval_method=SPLADE++_EnsembleDistil_ONNX --prompt_mode=rank_GPT  --context_size=4096 --variable_passages --window_size=20 --sglang_batched

pjyi2147 avatar Oct 09 '24 16:10 pjyi2147