MAE tests are missing in report

[emacha]

Overview

Some of the regression metrics (MAE and MeanError at least) are not being executed when running a report.

Example code

import pandas as pd
from evidently.future.datasets import Dataset, DataDefinition, Regression
from evidently.future.metrics import MAE, RMSE
from evidently.future.report import Report
from evidently.future import tests

data = pd.DataFrame({
    "id_col": [1, 2, 3],
    "actual": [10.0, 20.0, 30.0],
    "predicted": [11.0, 19.0, 29.0]
})
regression_ = [Regression(target="actual", prediction="predicted")]
data_definition = DataDefinition(id_column="id_col", regression=regression_)
dataset = Dataset.from_pandas(data, data_definition=data_definition)
report = Report(
    [
        # 3 tests for RMSE, 1 test for MAE -> Expected: 4 tests in total
        RMSE(tests=[tests.lt(100), tests.gt(0), tests.not_eq(0)]),
        MAE(tests=[tests.lt(100)]),
    ],
)
snapshot = report.run(dataset).dict()
assert len(snapshot["tests"]) == 4, "Expected 4 tests, but got {}".format(len(snapshot["tests"]))

>>> assert len(snapshot["tests"]) == 4, "Expected 4 tests, but got {}".format(len(snapshot["tests"]))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
>>> AssertionError: Expected 4 tests, but got 3

Versions

Evidently: 0.7.6 Python: 3.12.7

[elenasamuylova]

Hi @emacha,

MAE metric returns two different values (mean error and std), so when you create a test you'd need to use a different parameter depending in the value you are testing: mean_tests or std_tests.

# Updated imports

import pandas as pd

from evidently import Dataset, DataDefinition, Report, Regression
from evidently.metrics import MAE, RMSE
from evidently.tests import gt, not_eq, lt

# Data prep 
data = pd.DataFrame({
    "id_col": [1, 2, 3],
    "actual": [10.0, 20.0, 30.0],
    "predicted": [11.0, 19.0, 29.0]
})

regression_ = [Regression(target="actual", prediction="predicted")]
data_definition = DataDefinition(id_column="id_col", regression=regression_)

dataset = Dataset.from_pandas(data, data_definition=data_definition)

# Report

report = Report(
    [
        RMSE(tests=[lt(100), gt(0), not_eq(0)]),
        MAE(mean_tests=[lt(100)]), # use mean_tests or std_tests here
    ],
)
snapshot = report.run(dataset)

# Get output as dict
snapshot.dict()

[emacha]

@elenasamuylova Yeah that worked, thanks!

Do you think you could raise an exception in this case? It's a pretty large footgun.

[bharath03-a]

Hi @elenasamuylova! I’ve created a PR that raises an explicit exception when tests= is passed to MAE instead of mean_tests or std_tests. Happy to adjust for any suggested changes. Let me know your thoughts!