candle icon indicating copy to clipboard operation
candle copied to clipboard

Support for Microsoft Phi-3 128k context length

Open niutech opened this issue 1 year ago • 1 comments
trafficstars

Please allow to use microsoft/Phi-3-mini-128k-instruct model in the candle-phi example, which uses the LongRope scaling. Thanks!

niutech avatar Apr 24 '24 20:04 niutech

@niutech, we support 128k context length on mistral.rs which you can run with cargo run --release --features ... -- -i plain -m microsoft/Phi-3-mini-128k-instruct -a phi3.

For reference, here is our implementation: https://github.com/EricLBuehler/mistral.rs/blob/6334b30fdf6447fa787dcbedb032fb825c22ae1f/mistralrs-core/src/models/layers.rs#L84

I would be happy to contribute it.

EricLBuehler avatar Apr 25 '24 20:04 EricLBuehler