cloud-ai-sdk icon indicating copy to clipboard operation
cloud-ai-sdk copied to clipboard
verified publisher icon quic

Upgrade transformer library support

Open yh-yao opened this issue 1 year ago • 2 comments

The current SDK only supports "transformer=4.32.0". The newer version of transformer library supports a few new things (e.g. tokenizer.apply_chat_template, streamer). Upgrading the library will save a lot of time on implementing those things in the SDK repo.

yh-yao avatar May 14 '24 02:05 yh-yao

Hello yh-yao, There is no fix mapping rule that SDK should map to specific transformers version. The only concern is that Qualcomm's patch are used for specific transformers version. In general, we only add patch for src/transformers/modeling_outputs.py and specific modeling_xxx.py. So you can just patch it manually. Then you can use any transformers version.

dapengsmith avatar May 16 '24 00:05 dapengsmith

@dapengsmith I am trying to serve llama3-8b. It looks tricky to manually update the patching code. Since all of us want the Qualcomm SDK be used by more people, could you help me with updating the patch code?

yh-yao avatar May 20 '24 18:05 yh-yao

https://github.com/quic/efficient-transformers is now available for your LLM execution needs on Qualcomm AI 100 accelerators.

quic-aashwins avatar Jun 17 '24 21:06 quic-aashwins