Remove timm backbone weight loading to prevent meta-tensor warnings in `from_pretrained`

[HichTala]

What does this PR do?

This pull request makes a targeted change to the model loading logic in modeling_utils.py to improve compatibility with timm backbones. Specifically, it ensures that when a model is configured to use a timm backbone with pretrained weights (use_pretrained_backbone=True), the backbone weights are not loaded while the model is still in meta device during from_pretrained().

Model initialization and configuration:

• In from_pretrained, before model instantiation, if the config specifies both use_timm_backbone and use_pretrained_backbone, the code sets config.use_pretrained_backbone to False to defer loading pretrained timm backbone weights until after initialization.

Fixes # (issue)

• #37615

Before submitting

• [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
• [x] Did you read the contributor guideline, Pull Request section?
• [x] Was this discussed/approved via a Github issue or the forum? Please add a link to it if that's the case.
• [ ] Did you make sure to update the documentation with your changes? Here are the documentation guidelines, and here are tips on formatting docstrings.
• [ ] Did you write any new necessary tests?

Who can review?

@Rocketknight1 @qubvel @CyrilVallez @NielsRogge @xenova

[Cyrilvallez]

Hey @HichTala! Are you sure you are still facing the original issue in latest main? Loading has changed, and we do not use model.load_state_dict anymore!

[HichTala]

Hi @Cyrilvallez! I just tested this again with the latest main, and the original issue is still present.

The problem is not related to load_state_dict itself. It comes from the fact that during from_pretrained, when a model uses a timm backbone with use_pretrained_backbone=True, the model is still instantiated on meta devices, but the timm pretrained backbone weights are already being loaded at that stage. This causes all the warnings from timm, and then, once from_pretrained continues, those backbone weights are overwritten anyway by the Hugging Face checkpoint loading logic.

This PR simply removes loading the timm backbone weights until after the model is fully materialized in from_pretrained method.

[Cyrilvallez]

I see... Then this config switch should be hardcoded in all model config when this can happen, not in the general loading code! Do you want to make the change?

[HichTala]

Yep totally agree with that, I'm making the change!

[github-actions[bot]]

View the CircleCI Test Summary for this PR:

https://huggingface.co/spaces/transformers-community/circle-ci-viz?pr=42284&sha=18e083

[github-actions[bot]]

[For maintainers] Suggested jobs to run (before merge)

run-slow: conditional_detr, dab_detr, deformable_detr, detr, table_transformer