tract icon indicating copy to clipboard operation
tract copied to clipboard

Decluttering a `DequntizeLinearF32` followed by `QuantizeLinearI8` results in an error at model patching

Open mmagician opened this issue 1 year ago • 3 comments

Quantizing the first part of Tensorflow's model from this tutorial fails when trying to declutter: cargo run path-to-model/model_quant.onnx -i 1,784,u8

2024-01-10T13:33:10.956561599Z ERROR tract] ModelBuildingError

    Caused by:
        0: Error at stage declutter
        1: running pass declutter
        2: declutter node #1 "serving_default_flatten_input:0_dequant" DequantizeLinearF32
        3: Trying to substitute a 1,784,I8 by 1,784,U8.

The issue seems to be here:

if !original_fact.compatible_with(new_fact) {
    bail!("Trying to substitute a {:?} by {:?}.\n{:?}", original_fact, new_fact, self);
}

Since U8 is not "compatible" with an I8, the code bails out. Two very uninformed ideas would be to either:

  1. implement compatible_with for U8 <-> I8 (in fact, removing the compatibility check seems to work and replaces the two Dequant, Quant operations with a single LookupTable. I'm not sure whether that's the expected behaviour, but looking at the code I think yes).
  2. Call shunt_outside_unchecked instead of shunt_outside inside the if incoming_dt == DatumType::I8 || incoming_dt == DatumType::U8 condition, where we're guaranteed (?) to have u8/i8 only.

Without decluttering, all works fine:

cargo run path-to-model/model_quant.onnx -i 1,784,u8 --declutter-step 0

Resulting in:

┏ 0 Source serving_default_flatten_input:0
┃   ━━━ 1,784,U8
┣ 1 DequantizeLinearF32 serving_default_flatten_input:0_dequant
┃   ━━━ 1,784,F32
┣ 2 QuantizeLinearI8 tfl.quantize
┃   ━━━ 1,784,I8
┣ 3 DequantizeLinearF32 sequential/flatten/Reshape_dequant
┃   ━━━ 1,784,F32

mmagician avatar Jan 10 '24 13:01 mmagician