WhisperKit
WhisperKit copied to clipboard
MLX model support
Draft PR for the early stages of supporting MLX based whisper models directly in WhisperKit. (To be updated)
Initial TODOs:
- [x] Setup swift package structure
- [x] MLXFeatureExtractor using
FeatureExtracting
protocol - [x] MLXAudioEncoder inference using
AudioEncoding
protocol - [x] MLXTextDecoder inference using
TextDecoding
protocol orTextDecoder
subclass (research needed) - [x] Downloading MLX models (research needed to mix and match model downloads between CoreML and MLX)
- [x] Loading MLX models with a new model protocol
- [x] Pipeline allowing mixing CoreML and MLX models
- [x] ~Compiling whisperkit-cli optionally with/without MLX~ Crossing this out for now: whisperkit-cli does not have the same bundle size constraints as the core library
- [x] CI unit tests for iOS platform using
xcodebuild
instead of iOS simulator (simulator is not supported for MLX) - [ ] Review, cleanup, and documentation