icevision

icevision copied to clipboard

📓 New <Tutorial/Example>

Is this a request for a tutorial or for an example?
Tutorial

What is the task? Lets say I trained a model and now I want to check its performance on the brand new dataset. I don't want to retrain the whole model, I just want to check its performance. How to do it using Metric Class. Lets assume that COCO metric is fine for me.

metric = COCOMetric(metric_type=COCOMetricType.bbox)

How do I use it so I know exact performance on the new dataset?

---

Don't remove Main issue for examples: #39

Actually, there is no current easy way of doing this, can you describe the solution you would like?

Maybe a function that take the model, a dataloader, and the metrics?

Maybe a list of metrics instead of a single metric?

@fstroth @potipot you guys both worked on COCOMetric. Is this possible? Frederik, maybe with your dashboard?

@FraPochetti The dashboard lib already allows for evaluating a model on new (validation) data outside of training.

Great. Closing this one, then. @tugot17 please refer to the IceVision Dashboard built by @fstroth for more info.

There is an option to test a trained model. Assuming its weights are already loaded you can use for fastai:

learner.validate(test_dl) 

for pytorch-lightning

result = trainer.test(lightning_model, test_dl)

There is an option to test a trained model. Assuming its weights are already loaded you can use for fastai:

learner.validate(test_dl) 

for pytorch-lightning

result = trainer.test(lightning_model, test_dl)

Would this calculate the COCOMetric? I guess it would compute any metric the learner (or PL Module) was instantiated with.

I'd say this could be a FAQ so we should think of showcasing this in a notebook linked on a website.

Yes I agree. Keeping it here open as a reminder

@ai-fast-track maybe we can add a cell or two showing how to do this to the inference.ipynb

@potipot @FraPochetti

result = trainer.test(lightning_model, test_dl)

This does not work in PL right now since IceVision PL adapters do not implement the test_step (eg: https://github.com/airctic/icevision/blob/master/icevision/models/torchvision/lightning_model_adapter.py)

pytorch_lightning.utilities.exceptions.MisconfigurationException: No `test_step()` method defined to run `Trainer.test`

@tugot17 Just opened a PR to add support for Pytorch Lightning Trainer test with IceVision models.

Take a look https://github.com/airctic/icevision/pull/1125

This should solve your issue.