SDMetrics
SDMetrics copied to clipboard
`compute_metrics` for metrics with different signatures
Environment Details
Please indicate the following details about the environment in which you found the bug:
- SDMetrics version: 0.3.2
- Python version: 3.7
Error Description
When using sdmetrics.compute_metrics we get None for metrics that were not able to resolve keyword arguments. In most cases, we expect the user to pass a dictionary for metrics to compute_metrics but if the signature of these metrics differ, we get an error that causes us to catch it then store None in its location.
For example, take the following two time series metrics TSFCDetection and TSFClassifierEfficacy. The first one is expected to be called with
TSFCDetection.compute(data, sampled, entity_columns=entity_columns)
and the second one with an additional argument target
TSFClassifierEfficacy.compute(data, sampled, entity_columns=entity_columns, target=target)
compute_metrics will pass target to both classes which TSFCDetection cannot handle, thus crashing.
Steps to reproduce
import pandas as pd
from sdmetrics.timeseries import TSFCDetection, TSFClassifierEfficacy
length = 10
real = pd.DataFrame({
"seq_index": [1, 2, 3] * length,
"dim_0": [0, 0, 0] * length,
"dim_1": [4, 4, 4] * length
})
synth = pd.DataFrame({
"seq_index": [1, 2, 3] * length,
"dim_0": [1, 0, 0] * length,
"dim_1": [4, 4, 3] * length
})
metrics = {
'TSFClassifierEfficacy': TSFClassifierEfficacy,
'TSFCDetection': TSFCDetection
}
sdmetrics.compute_metrics(metrics, real, synth, entity_columns=['seq_index'], target='dim_0')