candle
candle copied to clipboard
Support for Microsoft Phi-3 128k context length
trafficstars
Please allow to use microsoft/Phi-3-mini-128k-instruct model in the candle-phi example, which uses the LongRope scaling. Thanks!
@niutech, we support 128k context length on mistral.rs which you can run with cargo run --release --features ... -- -i plain -m microsoft/Phi-3-mini-128k-instruct -a phi3.
For reference, here is our implementation: https://github.com/EricLBuehler/mistral.rs/blob/6334b30fdf6447fa787dcbedb032fb825c22ae1f/mistralrs-core/src/models/layers.rs#L84
I would be happy to contribute it.