Use of custom scoring function

[ardabeyazoglu]

Hi,

There is no information on docs about how to use a custom scorer. Here i tried this simple example without luck:

def customScorer(y_true, y_pred):
    score = y_pred['Signal'] == 1 and y_true['Rate'] > 0.5
    return score

model.train(
        train_data,
        model_names=['XGBClassifier'],
        train_uncertainty_model=['XGBClassifier'],
        cv=5,
        _scorer=make_scorer(customScorer, greater_is_better=True)
    )

Is there anything missing here ?

[ClimbsRocks]

great question! i'd forgotten all about this part of the dataset. it's worth noting too that this is just a scoring function, not an objective function that is used by the model at training time.

could you include the full stack trace? this feature's definitely still in alpha release (thus the no documentation), but would love your feedback on it!

we might only support a couple of pre-baked strings as scorers, like 'median_absolute_error' and 'accuracy'. ping me again tomorrow if you're still interested in this and i'll look into it more!