semantic-segmentation-ml-pipeline

semantic-segmentation-ml-pipeline copied to clipboard

Thought it would be good to have model cards in HfPusher. We can firstly discuss here, implement and later open a PR to TFA. I took a look at training script and thought it would be good to include: (to-do)

• [x] A template in case user feels like filling in later
• [ ] Hyperparameters
• [ ] Metrics
• [ ] Plot of the model
• [ ] A prepared text we can modify according to model configs (if user doesn't want to fill in later) (like @sayakpaul did with MAXIM models) We can also push training data in a dataset repository and link it to the model.

We can also push tensorboard logs BTW if you'd like to add it to training script. You can see how we host them here: https://huggingface.co/keras-io/lowlight-enhance-mirnet/tensorboard WDYT?

@merveenoyan thanks for initiating the conversation. Your proposal sounds solid to me.

Just for our context, https://huggingface.co/chansung/segmentation-training-pipeline/ - this is how the current repository looks like for the pushed model. The model is automatically pushed from a TFX pipeline (refer here and here).

Would be really good to include model cards here in the Hub model repository as huggingface_hub already provides a couple of neat utilities to do that.

It's also possible to customize the model card w.r.t the components mentioned by Merve (hyperparameters, metrics, dataset, etc.). A concrete example is here.

I believe the model card utilities should be implemented as a part of HFPusher().

@sayakpaul can you assign this to me? I'll create a tracker for model card parts above so we can handle it one step at a time and come up with something very fast.

@deep-diver could invite Merve to the repo? I will assign right away after it.

@sayakpaul

I just invited her as a collaborator to this repository!

About this example of using TensorBoard(https://huggingface.co/keras-io/lowlight-enhance-mirnet/tensorboard), can we automatically generate Model card based on the TensorBoard logs?

also, it is worth noting how TFX official ModelCardGenerator works and what it generates.

• https://github.com/digits/tfx-addons/blob/add-model-card-component/examples/model_card_generator/MLMD_Model_Card_Toolkit_Demo.ipynb

@deep-diver normally the tensorboard logs need to be pushed. the logs in model card are regardless of tensorboard logs but rather depend on model history, so if we keep history we can write metrics to card 🙂 it gets a bit exhausting though when there are too many epochs so we should have it under a toggle IMO. @deep-diver I was in the team building the model cards on Hub and I did look into the model cards toolkit itself and I find it a bit overkill for generating a simple markdown file. I think the model card they create includes nice evaluations for fairness but is too overwhelming, the model cards should be minimal-er compared to that one IMO. WDYT? @sayakpaul

We can respect the formats. I.e., if we're pushing to Hugging Face Hub, then I guess it's best to follow what's typically followed / suggested for the Hub model repositories as they've already gone through rigorous validation.

For Hub model cards here, I think it's best to combine the core eval metrics and whatever we get from history.

There are some things to note about TensorBoard.

• The logs of TensorBoard does not represent the final performance. It comes with many logs including lots of experiments.
  • I think a model card should contain only the final performance measurement on a separate dataset.
  • in TFX, the model is trained through the normal training process in Trainer component, and it produces ModelRun artifact which points to where the TensorBoard logs are if we have specified the TensorBoard callback.
  • in this case, we could pass ModelRun artifact to the HFPusher component so that HFPusher could build a model card based on TensorBoard logs.
• However, in TFX, the model is evaluated on Evaluator component whose sole purpose is to evaluate the trained model. It evaluates a model through many different eyes. For instance, how well does the model do on different classes instead of overall(average) performance?
  • TFX standard ModelCardGenerator creates a Model Card based on the output of Evaluator component. Like you said, it seems like overkill, but we could think it like we can take almost every reports based on our interests (may just need to filter). It also generates any types of output based on Jinja template engine such as Markdown or HTML, and we could even build our own template instead of the defaults ones.

I am just sharing what I know, so we could discuss and decide!

@merveenoyan, @sayakpaul

I got an idea for initial implementation.

1. first read TensorBoard logs into DataFrame [reference]
2. second create model card based on the first step using huggingface_hub utility package [reference]

I guess this is a good starting point, and we can research more about TensorFlow official model card toolkit and Evaluator component later. What do you guys think?

@deep-diver there's a very lightweight dependency called tabulate which we can use to turn TB logs in tables and host in model card. We can also keep it as CSV file separately if you wish.

Evaluator sounds good to me.