roc_auc

[sagol]

When I tried to use a metric roc_auc, I got an error: ValueError: Found input variables with inconsistent numbers of samples: [64, 128]

"train_batch_size": 64, "eval_batch_size": 64,

multi_label=False

[Pawel-Kranzberg]

Current version of roc_auc is for multi_label=True, AFAIK. You could use those instead:

from sklearn.preprocessing import LabelBinarizer
from sklearn.metrics import roc_auc_score

def multiclass_roc_auc_score_macro(y_pred:Tensor, y_true:Tensor, average = 'macro', sample_weight = None, **kwargs):
    # Based on https://medium.com/@plog397/auc-roc-curve-scoring-function-for-multi-class-classification-9822871a6659
    y_pred = np.argmax(y_pred, axis = 1)
    lb = LabelBinarizer()
    y_true = lb.fit_transform(y_true)
    y_pred = lb.transform(y_pred)
    return roc_auc_score(y_true, y_pred, average = average, sample_weight = sample_weight)

def multiclass_roc_auc_score_micro(y_pred:Tensor, y_true:Tensor, **kwargs):
    return multiclass_roc_auc_score_macro(y_pred, y_true, average = 'micro')

def roc_auc_score_by_class(y_pred:Tensor, y_true:Tensor, labels:list = labels_list, average = 'micro', sample_weight = None, **kwargs):
    y_pred = np.argmax(y_pred, axis = 1).numpy()
    y_true = y_true.detach().cpu().numpy()
    roc_auc_score_d = {}
    for i in range(len(labels)):
        lb = LabelBinarizer()
        y_true_i = y_true.copy()
        y_true_i[y_true != i] = len(labels) + 1
        y_true_i = lb.fit_transform(y_true_i)
        y_pred_i = y_pred.copy()
        y_pred_i[y_pred != i] = len(labels) + 1
        y_pred_i = lb.transform(y_pred_i)
        roc_auc_score_d[labels[i]] = roc_auc_score(y_true_i, y_pred_i, average = average, sample_weight = sample_weight)
    return roc_auc_score_d

The last one requires the list of labels - it needs to be provided before the function definition. All these functions have to be defined in your script/notebook before constructing the metrics list. I do it like that:

### Metrics functions
def multiclass_roc_auc_score_macro(y_pred:Tensor, y_true:Tensor, average = 'macro', sample_weight = None, **kwargs):
    # Based on https://medium.com/@plog397/auc-roc-curve-scoring-function-for-multi-class-classification-9822871a6659
    y_pred = np.argmax(y_pred, axis = 1)
    lb = LabelBinarizer()
    lb.fit(y_true)
    y_true = lb.transform(y_true)
    y_pred = lb.transform(y_pred)
    return roc_auc_score(y_true, y_pred, average = average, sample_weight = sample_weight)

# (...)

args = {
    'job_label': job_label,
    # (...)
    'FP16_loss_scale': 128,
    'metrics': {
                'functions': {
                        #'FastBert roc_auc': roc_auc,
                        'FastBert accuracy': accuracy,
                        #'FastBert fbeta': fbeta,
                        'confusion_matrix_overall': confusion_matrix_overall,
                        'confusion_matrix_by_class': confusion_matrix_by_class,
                        'multiclass_roc_auc_score_macro': multiclass_roc_auc_score_macro,
                        'multiclass_roc_auc_score_micro': multiclass_roc_auc_score_micro,
                        'roc_auc_score_by_class': roc_auc_score_by_class,
                        'F1_macro': F1_macro,
                        'F1_micro': F1_micro,
                        'F1_by_class': F1_by_class
                        }
            }
}
# (...)
### Metrics setup:
metrics = []
for k, v in args['metrics']['functions'].items():
    metrics.append({'name': k, 'function': v})

### Model training:
learner = BertLearner.from_pretrained_model(databunch, args['BERT_model'], metrics, device, logger,
                                            finetuned_wgts_path = None, # FINETUNED_PATH,
                                            warmup_proportion = args['warmup_proportion'],
                                            grad_accumulation_steps = args['gradient_accumulation_steps'],
                                            multi_gpu = multi_gpu,
                                            is_fp16 = args['FP16'],
                                            loss_scale = args['FP16_loss_scale'],
                                            multi_label = False)

[ahmedbahaaeldin]

where do we copy this code of the metrics ? confusion_matrix_overall is not defined and others