pytorch multipoint regression model cards

[mkurczew]

Hi! I went over the notebooks and some of the code for the Model Card Generator, but it seems that all three examples are classifiers.

What to do in case where I would like to create a model card for a multi-point regressor? For example, let's assume that we have a torch model that outputs 21 2-D facial landmarks. Let's say that I would like to show that NME(normalized mean error) is the same e.g., across all geos or races.

The make_eval_dataframe() method in model_card_gen/intel_ai_safety/model_card_gen/analyze/torch_analyzer.py assumes that there's a single numerical column representing labels and predictions so that probably wouldn't work for 21x2 numbers.

Any hints on how to tackle this kind of model with your toolkit?

[tybrs]

@mkurczew Thanks for the comment! We had a similar issue with multi-label support with pytorch. The workaround we received can be found here: https://github.com/tensorflow/model-analysis/issues/162

In short i think this could be solvable by creating a custom beam.Pipeline to generate evaluation results.