disent icon indicating copy to clipboard operation
disent copied to clipboard

[FEATURE]: Allow Metrics To Directly Accept Frameworks & Datasets

Open nmichlo opened this issue 3 years ago • 1 comments

Is your feature request related to a problem? Please describe. Current metrics require that you provide a representation function. This is inconvenient and always repeated. Metrics also always require that the dataset be a DisentDataset.

dataset = DisentDataset(data)
# we always need to produce this same function
get_repr = lambda x: module.encode(x.to(module.device))
# to use with metrics
results = metric_mig(dataset, get_repr)

Describe the solution you'd like Allow the metrics to directly accept the frameworks instead, and automatically wrap datasets with DisentDataset

# directly use the framework and raw data instead!
results = metric_mig(data, module)

Describe alternatives you've considered

  • Detect if isinstance(obj, DisentFramework) then handle everything automatically!
  • Or add a function to frameworks called get_representation
  • Or detect if an object has the property encode and use that instead!

nmichlo avatar Feb 06 '22 10:02 nmichlo

This will be easier now that metrics use the @metric decorator

nmichlo avatar Jun 09 '22 04:06 nmichlo