[FR] Mlflow Evaluation method to provide feasibility to subset the list of evaluation metrics

[khojarohan4]

Willingness to contribute

Yes. I would be willing to contribute this feature with guidance from the MLflow community.

Proposal Summary

Currently, based on the type of the model be it classification or regression, with evaluators as default, Mlflow evaluate method outputs a defined list of evaluation metrics. This feature request is to check if we could achieve a kind of filtering in the evaluation metric so the user can select which of the default metrics is required. In some case, not all metrics are required and this provides the user with more flexibility.

Motivation

What is the use case for this feature?

User to select the list of evaluation metrics from the list present when default evaluator is used.

Why is this use case valuable to support for MLflow users in general?

More flexibility

Why is this use case valuable to support for your project(s) or organization?

User can only look at the list of metrics that are required

Why is it currently difficult to achieve this use case?

Details

No response

What component(s) does this bug affect?

• [ ] area/artifacts: Artifact stores and artifact logging
• [ ] area/build: Build and test infrastructure for MLflow
• [ ] area/docs: MLflow documentation pages
• [ ] area/examples: Example code
• [ ] area/model-registry: Model Registry service, APIs, and the fluent client calls for Model Registry
• [ ] area/models: MLmodel format, model serialization/deserialization, flavors
• [ ] area/pipelines: Pipelines, Pipeline APIs, Pipeline configs, Pipeline Templates
• [ ] area/projects: MLproject format, project running backends
• [ ] area/scoring: MLflow Model server, model deployment tools, Spark UDFs
• [ ] area/server-infra: MLflow Tracking server backend
• [ ] area/tracking: Tracking Service, tracking client APIs, autologging

What interface(s) does this bug affect?

• [ ] area/uiux: Front-end, user experience, plotting, JavaScript, JavaScript dev server
• [ ] area/docker: Docker use across MLflow's components, such as MLflow Projects and MLflow Models
• [ ] area/sqlalchemy: Use of SQLAlchemy in the Tracking Service or Model Registry
• [ ] area/windows: Windows support

What language(s) does this bug affect?

• [ ] language/r: R APIs and clients
• [ ] language/java: Java APIs and clients
• [ ] language/new: Proposals for new client languages

What integration(s) does this bug affect?

• [ ] integrations/azure: Azure and Azure ML integrations
• [ ] integrations/sagemaker: SageMaker integrations
• [ ] integrations/databricks: Databricks integrations

[dbczumar]

Hi @khojarohan4. Thank you for your feature request; this sounds like a good idea. I think that adding a metrics configuration key to the evaluator_config dictionary for the default evaluator whose value is a list of metrics to compute seems like a good way to approach it, and we would be excited to review a pull request that implements this approach. Please let me know if you have any questions.

[dbczumar]

Hi @khojarohan4 , just following up here - are you still interested in contributing this feature?

[khojarohan4]

Hi @dbczumar , Apologies I havent had a chance to even look at the codebase yet. I'm really willing to and just been on a tight deadline. If its urgent, I would take a step back. Also, I havent contributed anytime so if you have any doc for the process or any reference, I can atleast take a stab but this isnt likely soon. Let me know.

[mlflow-automation]

@dbczumar Please reply to comments.