Model Evaluation, Diagnostics and MLFlow Registry

[louismagowan]

Description

This PR is a recreation of this one which I was told to close and restart due to git issues on the pymc-marketing repo.

It could be nice to have some standardised model evaluation and diagnostic functions added to pymc-marketing. Ideally they'd be formulated in a way that makes them easy to log in MLFLow later on. It would also be cool to build on top of the MLFlow module, to create a custom mlflow.pyfunc.PythonModel class to allow users to be able to register their models in the MLFlow registry. This would allow people to serve and maintain their MMMs more easily, and could help with MMM refreshes too.

Standard model metrics could include:

• Bayesian R2
• MAPE, RMSE, MAE
• Normalised RMSE and MAE (to allow comparisons across datasets and methodologies - in particular with Robyn models, who use NRMSE as one of their 2 key metrics) etc.

Diagnostic metrics could include:

• Step size, divergences
• LOOCV metrics e.g.

Model Registry Code:

A wrapper for an MMM model to make it conform to the MLFlow api, enabling registering and easier deployment I'll open this as a Draft for now - since I'll need advice on how where best to put this code, as well as overall design etc.

Related Issue

• [x] Closes #911
• [x] Closes #973
• [x] Related to #891 #901

Checklist

• [] Checked that the pre-commit linting/style checks pass
• [ ] Included tests that prove the fix is effective or that the new feature works
• [ ] Added necessary documentation (docstrings and/or example notebooks)
• [ ] If you are a pro: each commit corresponds to a relevant logical change

Modules affected

• [x] MMM
• [ ] CLV

Type of change

• [x] New feature / enhancement
• [ ] Bug fix
• [ ] Documentation
• [ ] Maintenance
• [ ] Other (please specify):

---

📚 Documentation preview 📚: https://pymc-marketing--1026.org.readthedocs.build/en/1026/