SDMetrics icon indicating copy to clipboard operation
SDMetrics copied to clipboard

`compute_metrics` for metrics with different signatures

Open sarahmish opened this issue 4 years ago • 0 comments

Environment Details

Please indicate the following details about the environment in which you found the bug:

  • SDMetrics version: 0.3.2
  • Python version: 3.7

Error Description

When using sdmetrics.compute_metrics we get None for metrics that were not able to resolve keyword arguments. In most cases, we expect the user to pass a dictionary for metrics to compute_metrics but if the signature of these metrics differ, we get an error that causes us to catch it then store None in its location.

For example, take the following two time series metrics TSFCDetection and TSFClassifierEfficacy. The first one is expected to be called with

TSFCDetection.compute(data, sampled, entity_columns=entity_columns)

and the second one with an additional argument target

TSFClassifierEfficacy.compute(data, sampled, entity_columns=entity_columns, target=target)

compute_metrics will pass target to both classes which TSFCDetection cannot handle, thus crashing.

Steps to reproduce

import pandas as pd
from sdmetrics.timeseries import TSFCDetection, TSFClassifierEfficacy

length = 10

real = pd.DataFrame({
    "seq_index": [1, 2, 3] * length,
    "dim_0": [0, 0, 0] * length,
    "dim_1": [4, 4, 4] * length
})

synth = pd.DataFrame({
    "seq_index": [1, 2, 3] * length,
    "dim_0": [1, 0, 0] * length,
    "dim_1": [4, 4, 3] * length
})

metrics = {
    'TSFClassifierEfficacy': TSFClassifierEfficacy,
    'TSFCDetection': TSFCDetection
}

sdmetrics.compute_metrics(metrics, real, synth, entity_columns=['seq_index'], target='dim_0')

sarahmish avatar Aug 25 '21 22:08 sarahmish