HuggingFace model/tokenizer loader
Is your feature request related to a problem? Please describe:
Some users might want to load model ckpt directly from HF to quickly explore and do some testing.
Describe the solution you would like:
Adding a wrapper class or a model family that create_model wrapping HF lib.
Describe the alternatives you have considered:
Simply expose a functional interface as a function but this is less ideal.
Additional Context:
For directly loading qwen models (inference) that are not implemented in fairseq2 yet.
Linking a relevant issue as this is partially complete:
https://github.com/facebookresearch/fairseq2/pull/1323
#1323
This is completed and has been tested with 1, 2, 8 devices.
https://github.com/facebookresearch/fairseq2/tree/rsy/hg-model-v1