Add push to hub functionality

[NielsRogge]

This PR adds a push_to_hub option to the YAML configuration. This enables people to push their trained model to the 🤗 hub, allowing easy inference afterwards. Moreover, each pushed model gets an automated model card along with a verl tag, enabling you to see which models people train using VeRL.

For PPO training:

python -m verl.trainer.main_ppo \
    ...
    trainer.push_to_hub=True \
    trainer.hub_model_id="username/model-name" \
    trainer.hub_private=False \
    trainer.hub_token="your_token" # Optional

For SFT training:

python -m verl.trainer.main_sft \
    ...
    trainer.push_to_hub=True \
    trainer.hub_model_id="username/model-name" \
    trainer.hub_private=False \
    trainer.hub_token="your_token" # Optional

Fixes #190

[vermouth1992]

Hi @NielsRogge, Nice work! Just wonder is there any CI that we can run to protect this functionality?

[NielsRogge]

Thanks for your comments, addressed them.

[NielsRogge]

Hi @eric-haibin-lin does this PR look good to you?

[CLAassistant]

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

[NielsRogge]

Hi @eric-haibin-lin just want to point out that we currently have 66k models on 🤗 tagged with TRL: https://huggingface.co/models?other=trl, and none for "verl".

Would be great to achieve the same for your library, if you're interested.

[eric-haibin-lin]

@NielsRogge we'd love to work and contribute to the huggingface community! huggingface invented great library and platform and is great to the community. But would the model_merger script be a better place since the safetensor conversion is done there? I'd suggest moving the hf tag / hf token changes to scripts/model_merger.py

[NielsRogge]

Yes, that could work. It would be nice if people can pass a boolean push_to_hf in the config so that the model gets pushed to the hub after training (and have "verl" in its metadata). Is this something you could open a PR for?