Can't convert Model generated with AutoQKeras

[LordScarface]

I was following the example in section 3 for automatically quantizing a Model using AutoQKeras. I then saved the best Model by adding qmodel.save('section3_qmodel_2.h5') at the end but now when I try to load that model and convert it using hls4ml I get the following error Message:

[...]
Interpreting Model
Topology:
Layer name: input, layer type: InputLayer, current shape: [[None, 28, 28, 1]]
Layer name: conv2d_0, layer type: QConv2D, current shape: [[None, 28, 28, 1]]
Traceback (most recent call last):
  File "/home/lukas/Documents/qkeras-mnist-example/qkeras_hls4ml.py", line 75, in <module>
    hls_model = hls4ml.converters.convert_from_keras_model(model,
  File "/home/lukas/.local/lib/python3.8/site-packages/hls4ml/converters/__init__.py", line 176, in convert_from_keras_model
    return keras_to_hls(config)
  File "/home/lukas/.local/lib/python3.8/site-packages/hls4ml/converters/keras_to_hls.py", line 313, in keras_to_hls
    layer, output_shape = layer_handlers[keras_class](keras_layer, input_names, input_shapes, reader, config)
  File "/home/lukas/.local/lib/python3.8/site-packages/hls4ml/converters/keras/qkeras_layers.py", line 96, in parse_qbatchnorm_layer
    layer['mean_quantizer'] = get_quantizer_from_config(keras_layer, 'mean')
  File "/home/lukas/.local/lib/python3.8/site-packages/hls4ml/converters/keras/qkeras.py", line 80, in get_quantizer_from_config
    return QKerasQuantizer(quantizer_config)
  File "/home/lukas/.local/lib/python3.8/site-packages/hls4ml/converters/keras/qkeras.py", line 11, in __init__
    self.alpha = config['config'].get('alpha', None)
TypeError: 'NoneType' object is not subscriptable

Looks like the first two Layers work fine, but it crashes when it gets to the QBatchNormalization Layer.

Here is the Code I use to convert the Model:

import tensorflow as tf
from tensorflow.keras.datasets import mnist
from tensorflow.keras.utils import to_categorical
from qkeras.utils import _add_supported_quantized_objects
import hls4ml
from sklearn.metrics import accuracy_score
import numpy as np

co = {}
_add_supported_quantized_objects(co)
model = tf.keras.models.load_model('./section3_qmodel_2.h5', custom_objects=co)

config = hls4ml.utils.config_from_keras_model(model, granularity='name')

hls_model = hls4ml.converters.convert_from_keras_model(model,
                                                       hls_config=config,
                                                       output_dir='/home/lukas/Documents/qkeras-mnist-example/hls_test_prj',
                                                       fpga_part='xczu7ev-ffvc1156-2-e')

hls4ml.utils.plot_model(hls_model, show_shapes=True, show_precision=True, to_file='./qmodel_3.jpg')

hls_model.compile()

I'll also attach my saved Model: section3_qmodel_2.zip

[vloncar]

Hi Lukas (LordScarface), Thanks for your interest in hls4ml. I see two problems here, first is that QBatchNormalization is created without any quantizers, which I could say is the issue with AutoQ, however we can handle it better in hls4ml. With some defensive code after this line we can get further with the parsing, however later in the model you use quantized_relu_po2 activation, which is not yet supported in hls4ml.

[LordScarface]

Hi and thank you for your Help!

I just now got around to this, but for testing I decided to not quantize the QBatchNormalization Layer (use regular BatchNormalization Layer instead) and I also replaced the quantized_relu_po2 activation with regular quantized_relu Activations. Now when I load the Model hls4ml translates it just fine, but the accuracy is degraded (~71% on the hls_model vs. ~96% on the Keras Model). There might be something wrong with my configuration, because when I call hls4ml.model.profiling.numerical(...) I get

[...]
Creating HLS model
Profiling weights (before optimization)
Weights for conv2d_0 are only zeros, ignoring.
Weights for conv2d_1 are only zeros, ignoring.
Weights for conv2d_2 are only zeros, ignoring.
Weights for conv2d_3 are only zeros, ignoring.
Weights for dense are only zeros, ignoring.
Profiling weights (final / after optimization)
Weights for conv2d_0 are only zeros, ignoring.
Weights for conv2d_1 are only zeros, ignoring.
Weights for conv2d_2 are only zeros, ignoring.
Weights for conv2d_3 are only zeros, ignoring.
Weights for dense are only zeros, ignoring.
Weights for dense_alpha are only zeros, ignoring.
[...]

This is the Code I use to generate the HLS Project:

import tensorflow as tf
from tensorflow.keras.datasets import mnist
from tensorflow.keras.utils import to_categorical
from qkeras.utils import _add_supported_quantized_objects
import hls4ml
from sklearn.metrics import accuracy_score
import numpy as np
from sklearn.datasets import fetch_openml
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import LabelEncoder, StandardScaler

co = {}
_add_supported_quantized_objects(co)

def get_train_test_set():
    (x_train, y_train), (x_test, y_test) = mnist.load_data()
    
    x_train = x_train.reshape(x_train.shape + (1,))
    x_test = x_test.reshape(x_test.shape + (1,))
    
    x_train = x_train / 256.0
    x_test = x_test / 256.0
    
    x_mean = np.mean(x_train, axis=0)

    x_train -= x_mean
    x_test -= x_mean
        
    y_train = to_categorical(y_train, 10)
    y_test = to_categorical(y_test, 10)
    
    return (x_train, y_train), (x_test, y_test)

def print_dict(d, indent=0):
    align=20
    for key, value in d.items():
        print('  ' * indent + str(key), end='')
        if isinstance(value, dict):
            print()
            print_dict(value, indent+1)
        else:
            print(':' + ' ' * (20 - len(key) - 2 * indent) + str(value))
            
(x_train, y_train), (x_test, y_test) = get_train_test_set()

model = tf.keras.models.load_model('./section3_qmodel_2.h5', custom_objects=co)
print(model.summary())

cfg = hls4ml.utils.config_from_keras_model(model, granularity='name')

hls4ml.model.optimizer.OutputRoundingSaturationMode.layers = ['Activation']
hls4ml.model.optimizer.OutputRoundingSaturationMode.rounding_mode = 'AP_RND'
hls4ml.model.optimizer.OutputRoundingSaturationMode.saturation_mode = 'AP_SAT'

print_dict(cfg)

hls_model = hls4ml.converters.convert_from_keras_model(model,
                                                       hls_config=cfg,
                                                       output_dir='section3_hls4ml_prj_2',
                                                       fpga_part='xczu7ev-ffvc1156-2-e')
hls_model.compile()

hls4ml.model.profiling.numerical(model=model, hls_model=hls_model, X=x_test[:1000])
hls4ml.utils.plot_model(hls_model, show_shapes=True, show_precision=True, to_file='section3_qmodel_2.jpg')

y_hls = hls_model.predict(x_test)
y_qke = model.predict(x_test)

print("QKeras Accuracy: {}".format(accuracy_score(np.argmax(y_test, axis=1), np.argmax(y_qke, axis=1))))
print("hls4ml Accuracy: {}".format(accuracy_score(np.argmax(y_test, axis=1), np.argmax(y_hls, axis=1))))

I'll also attach the Model I am using.