addons icon indicating copy to clipboard operation
addons copied to clipboard

Metrics don't work with categorical / one-hot outputs

Open lnicola opened this issue 4 years ago • 5 comments

Apologies, this might actually be my egregious lack of experience in this domain.

I've set up a little network to play with:

# xs_training.shape is (255722, 6, 7)
# y_validation.shape is (694001,)
model = Sequential([
  Conv1D(128, 3, activation='relu', padding='same', input_shape=(xs_training.shape[1], xs_training.shape[2])),
  MaxPooling1D(2, padding='same'),
  Conv1D(256, 3, activation='relu', padding='same'),
  MaxPooling1D(2, padding='same'),
  Flatten(),
  Dense(256, activation='relu'),
  Dropout(0.2),
  Dense(256, activation='relu'),
  Dropout(0.2),
  Dense(len(d), activation='softmax'),
])
model.summary()
model.compile(
  optimizer=Nadam(),
  loss='categorical_crossentropy',
  metrics=['accuracy', CohenKappa(len(d))],
)
model.fit(
  xs_training,
  to_categorical(y_training),
  epochs=10,
  batch_size=64,
  validation_data=(xs_validation, to_categorical(y_validation))
)

This works if I only use accuracy as a metric, but if I add CohenKappa I get:

NotImplementedError: in user code:

    /usr/lib/python3.9/site-packages/tensorflow/python/keras/engine/training.py:805 train_function  *
        return step_function(self, iterator)
    /home/grayshade/.local/lib/python3.9/site-packages/tensorflow_addons/metrics/cohens_kappa.py:205 result  *
        weight_mtx = tf.ones([nb_ratings, nb_ratings], dtype=tf.float32)
    /usr/lib/python3.9/site-packages/tensorflow/python/util/dispatch.py:201 wrapper  **
        return target(*args, **kwargs)
    /usr/lib/python3.9/site-packages/tensorflow/python/ops/array_ops.py:3120 ones
        output = _constant_if_small(one, shape, dtype, name)
    /usr/lib/python3.9/site-packages/tensorflow/python/ops/array_ops.py:2804 _constant_if_small
        if np.prod(shape) < 1000:
    <__array_function__ internals>:5 prod
        
    /usr/lib/python3.9/site-packages/numpy/core/fromnumeric.py:3030 prod
        return _wrapreduction(a, np.multiply, 'prod', axis, dtype, out,
    /usr/lib/python3.9/site-packages/numpy/core/fromnumeric.py:87 _wrapreduction
        return ufunc.reduce(obj, axis, dtype, out, **passkwargs)
    /usr/lib/python3.9/site-packages/tensorflow/python/framework/ops.py:852 __array__
        raise NotImplementedError(

    NotImplementedError: Cannot convert a symbolic Tensor (strided_slice_1:0) to a numpy array. This error may indicate that you're trying to pass a Tensor to a NumPy call, which is not supported

I suppose the problem is that CohenKappa doesn't work with one-hot vectors (Accuracy somehow detects them and calls CategoricalAccuracy, doesn't it?).

lnicola avatar Apr 26 '21 16:04 lnicola