Disparate Impact Remover doesn't seem to be working

[akuzni2]

I'm trying to run the disparate impact remover on a dataset - however after varying the repair value between 0 -> 1.0 I see no change in disparate impact metric.

Example: My dataset (df_encoded) looks like the below with around 600 samples.

dataset_train = StandardDataset(df_encoded, label_name='outcome',
                    favorable_classes=[0],
                    protected_attribute_names=['sensitive'],
                    privileged_classes=[[1]])

di_remover = DisparateImpactRemover(repair_level=1.0)
train_repd = di_remover.fit_transform(dataset_train)

privileged_groups = [{'sensitive': 1}]
unprivileged_groups = [{'sensitive': 0}]
    
metric_train_repd = BinaryLabelDatasetMetric(train_repd, 
                unprivileged_groups=unprivileged_groups,
                privileged_groups=privileged_groups)

explainer = MetricTextExplainer(metric_train_repd)
explainer.disparate_impact()

In my dataset i get a disparate impact of 0.71. When I change "repair_level" to 0 or 1.0 I see no change in disparate impact.

[cjwillis48]

Was a solution found for this? I am seeing the same thing now

[OnTheThirdDay]

I think that is actually the intended result. You may compare the dataset before and after, and the disparate_impact_remover has adjusted the feature values.

[pradeepdev-1995]

@OnTheThirdDay How can I compare the datasets( original dataset and the disparate impact removed dataset) through code?

[OnTheThirdDay]

@OnTheThirdDay How can I compare the datasets( original dataset and the disparate impact removed dataset) through code?

Hi @pradeepdev-1995 , I haven't followed this thread for long, so a bit forgot the context. To compare, you can export the dataset (e.g. into csv) and you can see the feature values are actually different, though the DI score is not changed.

[pradeepdev-1995]

di_remover = DisparateImpactRemover(repair_level=1.0)
train_repd = di_remover.fit_transform(dataset_train)

How can I export the train_repd to csv? kindly share the code snippet for this.

[OnTheThirdDay]

di_remover = DisparateImpactRemover(repair_level=1.0)
train_repd = di_remover.fit_transform(dataset_train)

How can I export the train_repd to csv? kindly share the code snippet for this.

Use some api like .convert_to_dataframe() and to_csv()

[pradeepdev-1995]

@OnTheThirdDay Checked with convert_to_dataframe function of AIF360 like this

di = DisparateImpactRemover(repair_level=1.0)
dir_dataset = di.fit_transform(dataset)
dir_dataset_df= dir_dataset.convert_to_dataframe(sep=",")

print(originalDataset.equals(dir_dataset_df[0]))

But it shows that original and transformed datasets are the same. Which means DisparateImpactRemover nothing changed in the dataset even the repair_level=1.0