[FEATURE] Toy model for testing

[adamjstewart]

Is your feature request related to a problem? Please describe.

In TorchGeo's unit tests, we want to avoid any large models that require a lot of memory or time to run.

Describe the solution you'd like

I would like to have a toy model with only 1 or 2 layers that runs as quickly as possible while still ensuring compatibility with other timm models.

Describe alternatives you've considered

We're currently writing our own fake models and monkeypatching timm but there are many discrepancies and it isn't the best test.

[adamjstewart]

Alternatively, what is the smallest model currently in timm?

[rwightman]

@adamjstewart this one https://github.com/huggingface/pytorch-image-models/blob/main/results/benchmark-infer-amp-nchw-pt113-cu117-rtx3090.csv#L2 .. tinynet_e

[rwightman]

@adamjstewart if you need even smaller we could figure out a model to add an even smaller one to, if it needs weights with an imagenet classifier would need to train

[adamjstewart]

That's still fairly large. I don't think we need pre-trained weights, we don't want to download anything during testing and most of our images are multispectral anyway.

[adamjstewart]

@rwightman any more thoughts on this? Still hoping for an incredible simple 1 or 2 layer model intended only for testing purposes.

[rwightman]

@adamjstewart So, 1-2 layer models don't make sense as it breaks the model APIs and all assumptions re feature maps, etc. I have made some much smaller models in the 300-400K param range, they have valid classifiers and something I can use for regression tests, proxies for full sized models meaninfully. So hopefully a bit better for your use case but not exactly what you want. Will be merged with some other changes soon...

e.g. https://huggingface.co/timm/test_vit.r160_in1k

model	top1	top1_err	top5	top5_err	param_count	img_size	crop_pct
test_efficientnet.r160_in1k	47.156	52.844	71.726	28.274	0.36	192	1.0
test_byobnet.r160_in1k	46.698	53.302	71.674	28.326	0.46	192	1.0
test_efficientnet.r160_in1k	46.426	53.574	70.928	29.072	0.36	160	0.875
test_byobnet.r160_in1k	45.378	54.622	70.572	29.428	0.46	160	0.875
test_vit.r160_in1k	42.0	58.0	68.664	31.336	0.37	192	1.0
test_vit.r160_in1k	40.822	59.178	67.212	32.788	0.37	160	0.875

[rwightman]

Another thing that could be done within test modules, create & register own defs...

e.g. If one had a unit test module, within that can define test specific models outside timm and register and use via timm API. Of course again many models have constraints in terms of needing at least 4 feature stages, channel counts predefined, etc. But some can be finaggled around that or have more configurability...

import timm

default_cfgs = timm.models.generate_default_cfgs({'mytest.blah': timm.models.resnet._cfg(url='http:/blah.com/myweights.pth', num_classes=10)})

@timm.models.register_model
def mytest(pretrained=False, **kwargs):
    from timm.models.resnet import _create_resnet, Bottleneck
    model_args = dict(
        block=Bottleneck, layers=(1, 1, 1, 1), channels=(8, 16, 32, 64), stem_width=8, stem_type='deep', replace_stem_pool=True,
        avg_down=True)
    return _create_resnet('mytest', pretrained, **dict(model_args, **kwargs))

mm = timm.create_model('mytest')

[adamjstewart]

This looks great. Still working on bumping SMP to support modern timm, and don't necessarily want to require the latest version of timm yet, but I will definitely come back to this someday and speed up all of our tests with smaller models.

[rwightman]

@adamjstewart if you need any points re 'modern timm' let me know, for the most part, especially if it was mostly just models being used and not importing less used modules / layers directly it should be really straight forward and I did lots of regression checks.

[adamjstewart]

I think we've got it mostly figured out, but I'll let you know if we get stuck anywhere.