keras-team
A separate page to track pre-trained models and their details
Once we have a collection of pre-trained models we can start preparing a Markdown page including the following details for each model:
(Note that I'm only assuming image classification models for now but these details could be amended to accommodate detection, and segmentation models too):
- Top-1 accuracy
- Top-5 accuracy
- Number of parameters
- FLOPs
- Number of images / sec (inference mode) (throughput)
- Contributor
- Notes on the benchmark hardware (this should ideally be the same for all the models for consistency, I suggest an A100)
For deriving the number of parameters, FLOPs, and throughput we could set up a job or a script for encouraging reproducibility.
Cc: @LukeWood
Absolutely! This will be fantastic. Ideally, our scripts will autogenerate all of these metrics and export them to either JSON, markdown, some other format! Then, keras.io can automatically load these at compile time and generate a nice leaderboard
For deriving the number of parameters, FLOPs, and throughput we could set up a job or a script for encouraging reproducibility.
Having tools for this in Keras It is one of the most popular FR of the last years: https://github.com/tensorflow/tensorflow/issues/32809
