hls4ml

hls4ml copied to clipboard

Hi all,

I'm having trouble while compiling a model similar to this

model = tf.keras.models.Sequential([
        tf.keras.layers.Input((10)),
        tf.keras.layers.Dense(10*3),
        tf.keras.layers.Reshape((10,3)),
        tf.keras.layers.Softmax()
])

These are the errors:

My guess is that the problem is someway related to the combination of Reshape + softmax. Thanks in advance for the help!

@vloncar I was wondering if this would be solved by #384 but the bug seems to be still present. I think the issue is that Reshape is handled by just an InPlaceVariable, which should refer to layer2_out, but somehow the (nonexistent) layer3_out is used as input to softmax.

Seeing that #384 has been merged and I'm still seeing this problem with a similar model construction, I don't think that was the solution. That said, I also arrived at the same general area of the code by noticing the InPlaceVariable definition in the reshape handler class while debugging this problem.

Prior to commit 7f75add50a5acd2a4335bde0ab98c9d4e79e1137 I was able to build the model, but that's because one of the optimization passes was replacing the Reshape with a Repack, which doesn't seem to have the same problem.

I'm going to see if I can make some sense as to what's going on. At a first glance, every other handler appears to be using self.add_output_variable instead of calling self.model.register_output_variable explicitly. Then again, Reshape is the only one using InplaceVariable, at least in the hls4ml/model/hls_layers.py file.

I'm not sure, but I think the problem is that for InplaceVariable, method variable_definition_cpp in hls4ml/writer/vivado_writer.py returns None, and method function_cpp in hls4ml/model/hls_layers.py returns None. In other words, nothing is output for this layer in the project C++ file.

As far as I can tell, the same is true for other forms of output, as in nothing is being generated. That leads me to the question: what does an InplaceVariable correspond to in the generated code? Is it a function call? Some kind of variable? Both? In my case, the Reshape layer is really a flatten operation, so I would expect some kind of assignment or function call that, well, flattens the output of the previous layer and forwards it to the next layer.

InplaceVariables can get messed up by optimizer passes. I fixed the issue in a branch I am working on, but I can try to extract it and make a pull request of just the fix later today.

Fixed by #468