WhisperKit icon indicating copy to clipboard operation
WhisperKit copied to clipboard

MLX model support

Open ZachNagengast opened this issue 10 months ago • 1 comments

Draft PR for the early stages of supporting MLX based whisper models directly in WhisperKit. (To be updated)

Initial TODOs:

  • [x] Setup swift package structure
  • [x] MLXFeatureExtractor using FeatureExtracting protocol
  • [x] MLXAudioEncoder inference using AudioEncoding protocol
  • [x] MLXTextDecoder inference using TextDecoding protocol or TextDecoder subclass (research needed)
  • [x] Downloading MLX models (research needed to mix and match model downloads between CoreML and MLX)
  • [x] Loading MLX models with a new model protocol
  • [x] Pipeline allowing mixing CoreML and MLX models
  • [x] ~Compiling whisperkit-cli optionally with/without MLX~ Crossing this out for now: whisperkit-cli does not have the same bundle size constraints as the core library
  • [x] CI unit tests for iOS platform using xcodebuild instead of iOS simulator (simulator is not supported for MLX)
  • [ ] Review, cleanup, and documentation

ZachNagengast avatar Apr 23 '24 04:04 ZachNagengast