Difference between act(y) and act(yref) causes errors when the activation function is LeakyRelu

[iotg-ba-hw-acceleration]

Hi, Inside the class SigmaDeltaNeuronsInstErrExec, the use of "self.process.act_fn(y) - self.process.act_fn(vars.y_ref)" to compute delta_a causes errors when the activation function is LeakyReLU. This happens because, delta_a is used to update y_ref. Consider a time t when the value of y becomes negative and the slope of leakyrelu for negative values be 0.1, the value of delta_a correctly corresponds to a value (y(t)0.1 - y_ref(t)) and it update y_ref(t) to reflect the value of 0.1y. But in the next time step when y_ref is passed to the activation function, the output becomes 0.01*y(t) and the absolute value of delta_a is more. Instead of applying the activation function over y_ref, a direct subtraction is sufficient and it works for ReLU and other non-linear activation functions also. I have added the section of code which worked for me.

delta_a = tf.cond( tf.logical_and( tf.greater_equal(self.time, self.process.first_valid_ts), tf.equal(tf.math.mod(self.time - self.process.first_valid_ts + 1, self.process.out_decim_interval), 0)), true_fn=lambda: self.process.act_fn(y) - vars.y_ref, false_fn=lambda: tf.zeros_like(vars.delta_a))