aequitas
aequitas copied to clipboard
Better handling of experiment output folders
Currently, when we run an experiment, the output is stored in a folder that is defined by the name
attribute in Experiment
. If we run the same experiment again with the same output folder, the folder is partially overwritten, which can lead to confusing situations.
Example: I ran an experiment for sensitive attribute A
with four categories: 1
, 2
, 3
and 4
. So, 4 different folders were created for each of those categories. Then, I ran the same experiment with sensitive attribute B
with two categories, 1
and 2
. Since the output folder name was the same for both experiments, the second experiment overwrote the folders for categories 1
and 2
but retained the categories 3
and 4
from the last experiment. This led to incorrect plots when plotting the results.
Ideally, we should have an attribute overwrite
that's False
by default. If it's False
, the experiment should throw an error if the given folder name already exists. If it's True
, we should delete the contents of the output folder before running the experiment to prevent data from two experiments being mixed up.