intel-extension-for-pytorch icon indicating copy to clipboard operation
intel-extension-for-pytorch copied to clipboard
verified publisher icon intel

Make key optional in ipex.llm.functional.rotary_embedding

Open sarckk opened this issue 7 months ago • 7 comments

Context: vLLM has a PR to make key arg in rotary embedding optional (https://github.com/vllm-project/vllm/pull/17566). Currently, this is not supported in ipex.llm.functional.rotary_embedding_batched that vLLM uses. I couldn't find rotary_embedding_batched here, so I'm assuming this is generated from rotary_embedding.

I wasn't able to build and test as I don't have access to intel cpu or xpu, looking for review/feedback. thanks

Note: this only modifies ipex.llm.functional.rotary_embedding API

sarckk avatar May 02 '25 00:05 sarckk

format, also wondering how do we test here?

houseroad avatar May 02 '25 00:05 houseroad

@jingxu10 @ZailiWang could you take a look?

sarckk avatar May 05 '25 17:05 sarckk

cc @zhuyuhua-v

jianan-gu avatar May 12 '25 06:05 jianan-gu

There is no CI in public repo. @jingxu10 @ZailiWang could you please cherry pick the PR and test it in internal repo?

tye1 avatar May 12 '25 08:05 tye1

There is no CI in public repo. @jingxu10 @ZailiWang could you please cherry pick the PR and test it in internal repo?

OK, let me do it and feedback the result.

ZailiWang avatar May 12 '25 08:05 ZailiWang

Hi @sarckk , thanks a lot for the contribution as well as the notification of recent updates in vLLM, we are implementing some other updating requests for vLLM integration, and this PR, along with some other changes needed at the frontend, will be merged then.

ZailiWang avatar May 21 '25 01:05 ZailiWang

@ZailiWang sounds good! thanks for closing the loop on this

sarckk avatar May 22 '25 05:05 sarckk