autogluon
autogluon copied to clipboard
Add more custom eval metric options to the MM module.
Adding a custom eval metric to the MM predictor is almost impossible. It's a link to the torchmetrics and there it's a way doing a custom torchmetric funtions, which then should be added to the predictor. I don't think many will succeed doing this without use many hours of work. The torchmetric has also limited default metrics to use. How about adding additional other possible metrics, such as the one from the tabular predictor function which is compatible with common codes, even if it's not equal as fast and optimized as the torchmetric.
I think we allow customized eval_metric
, but not an easy way of customizing the validation_metric
(a different concept). The main logic is here https://github.com/autogluon/autogluon/blob/22ea482e52067b4958f897b4628079249bc4f583/multimodal/src/autogluon/multimodal/utils/metric.py#L45-L129 . We need some documentation in how to customize the eval_metric
though.
I would agree that adding compatibility with Tabular's metrics will be a good feature addition, and will enable adopting Tabular's custom metric support logic. We should prioritize this for v0.8 release, as hopefully it isn't overly complicated to do.
Yes I have played with the code logic above before and read the documentation in torchmetric, read many examples etc etc. Fast and optimized when using default metrics but a new code language when doing a custom metric, and then one should as implement it here, so it's a two step work. So it sound good if the common way implementing a custom metric also could be an option, also good for benchmarking as many uses the sklearn versions.