Add support for quantization and custom audio context size to OpenVino

[dscripka]

Compiling with OpenVino on supported platforms can significantly improve performance, but currently it lacks two other features that can also significantly improve performance:

• Using quantized models (10%+ performance increase)
• Supporting custom audio context sizes (~3x performance increase for short audio files)

This PR adds both of these as optional arguments to the OpenVino model conversion scripts, enabling performance of quantization, audio context size, and OpenVino to stack.

There are a few important caveats, however:

• The OpenVino encoder (to my knowledge) only supports a fixed audio context size, so the converted model is somewhat more restricted
• This does require monkey patching a method from the openai-whisper library in convert-whisper-to-openvino.py, which isn't ideal
• Quantization is done with nncf 2.7.0, which currently only supports 4 and 8 bit quantization

Despite these, the performance improvement can be so substantial for certain use cases it may be worth it. For example, on the ~10 second jfk.wav file on a Intel(R) Xeon(R) W-2123 CPU:

Threads	Quant	Model	Encoder Time (s)	Arguments	Build
1	8 bit	small-en	8	-bs 1 -ac 1500	BLAS = 1
1	8 bit	small-en	0.8	-bs 1 -ac 550	OPENVINO = 1

Command to produce the OpenVino model for this PR: python convert-whisper-to-openvino.py --model small.en -ac 550 -qb 8

[jason-ni]

Is there a way to improve openvino encoder to support dynamic audio_ctx? That would be much more usefull.

[dscripka]

I agree that would be ideal, but it would require more substantial modifications to how the encoder is converted and the actual OpenVino runtime implementation. Possible perhaps, but difficult.

[jason-ni]

But CPU and other GPU backends all support dynamic audio_ctx given fixed model encoding size. If we allow converting OpenVino model with different encoding size, it would make model users confused.