Quantus icon indicating copy to clipboard operation
Quantus copied to clipboard

Enhancement: out metrics' **kwargs in *args and set default values

Open annahedstroem opened this issue 3 years ago • 1 comments

To make the parametrisation of metrics in Quantus more explicit (in order to mitigate the risks of user typos and other undefined behaviours e.g., when passing a larger kwargs for different perturb_func calls if large-scale experiments are performed), we want to update the arguments list of all metrics initialisations like def __init__(self, ...) in the library as follows:

  • Split kwargs for every Callable used in the metric init like explain_func, perturb_func and similar_func as follows: explain_func_kwargs, perturb_kwargs and similar_func_kwargs
  • Make sure to set default values to None where types are immutable in arguments list and reset it after to its actual data type like perturb_kwargs: Union[None, dict] = None
  • Remove *argssince it is useless

AND:

  • Update the docstrings to reflect these changes
  • Update code in the library where the metrics are called e.g., in the tutorials and pytests

annahedstroem avatar Mar 17 '22 12:03 annahedstroem

Merging with another issue: https://github.com/understandable-machine-intelligence-lab/Quantus/issues/78.

Quantus is designed to be easy to use: users are allowed to initialise the metrics without explicitly setting any metrics arguments (there are default params for everything) - which means that metrics can be easily looped over etc, making the library very flexible.

But since "the devil is in the details" when it comes to metric parameterisation - we should create a feature that highlights the current parameter setting of each metric - so to identify potential typos etc.

Could be a simple feature that prints the values of metrics' argument at metric initialisation, suggestively:

  • create a "disable_parameter_printing" function that checks for deviation of default values

annahedstroem avatar Apr 14 '22 14:04 annahedstroem

Completed in #124

annahedstroem avatar Sep 25 '22 15:09 annahedstroem