nncf
nncf copied to clipboard
openvinotoolkit
question about insert point
Hello, I use NNCF recently and I am wondering, if it is possible to insert a fake quantizer in a self-defined position, for example:
- fake quantizer after concat layer
- fake quantizer before and after resize layer
- fake quantizer before output layer
looking forward to your reply, many thanks！
Greetings, @jylansesuifeng!
NNCF currently does not provide a user-facing interface to put fake quantizers into arbitrary locations. The reasoning for this is that the best-performing model actually has to have all inputs of all of its low-precision compatible nodes quantized, and an arbitrary fake quantizer setup may actually end up quantizing only half of an operation's inputs due to non-locality of the fake quantizer's effect on downstream ops. We therefore primarily concern ourselves with quantizing all inputs of the required operations when preparing a quantizer setup, and not the outputs. This is done in the following manner - all operations that must have their inputs quantized have FQs placed directly at their inputs first, and then all such FQs are propagated upwards in the model's control flow graph.
You may set up "ignored_scopes" in the quantization algo's config parameters; adding an op to "ignored_scopes" will mean that a) if the op is inputs-quantizable, then no FQs will actually be added to the op's inputs to propagate upwards, and b) if the op is input-agnostic, such as maxpool or concat, then the downstream FQs will not propagate upwards through the associated operation. If you craft the "ignored_scopes" correctly, you may achieve the control you desire over the resulting FQ setup.
Alternatively, you may try to use nncf.quantization.algo.ExperimentalQuantizationBuilder which accepts nncf.quantization.quantizer_setup.SingleConfigQuantizerSetup structures that define a quantizer setup on a per-quantizer per-location basis and constructs a model/controller pair that correspond exactly to this given quantizer setup. You would have to construct the nncf.quantization.quantizer_setup.SingleConfigQuantizerSetup object yourself - see examples in the NNCF library code. Also, these are not yet parts of public NNCF API and so are subject to change without notice, so I would only advise you to use this path for experimentation.
I put fake quantizers into arbitrary locations via pattern-based quantization, however, which has been removed in the last NNCF.
I would be appreciated it if nncf could provide some flexible APIs to add custom FQ for the propagation-based quantization. @vshampor
I put fake quantizers into arbitrary locations via pattern-based quantization, however, which has been removed in the last NNCF.
I would be appreciated it if nncf could provide some flexible APIs to add custom FQ for the propagation-based quantization. @vshampor
@vshampor It's been a long while since the last comment; Has the suggested API been implemented? If not, can we make an issue that requires someone to make an API for creating custom Fake Quantizers? If yes, I'd like to try and make that API, doesn't sound too complicated.
