[Proposal] Modular config for RL (initially for Skrl lib)

[lopatovsky]

Proposal

Implement Reinforcement Learning config class configuring all parameters related to RL ( e.g. RL agent, RL model. ) utilizing modular fashion identical to ManagerBasedRLEnvCfg architecture.

This is an example how it could be used : (The example is showing the main config as it could be used for Humanoid task with some simple overwrites. Subconfigs are declared elsewhere with default values taken from agents/skrl_ppo_cfg.yaml)

@configclass
class HumanoidRLCfg(RLCfg):

    seed = 42
    rl_library = "skrl"

    model: ModelCfg = SkrlModelCfg(
        separate=True,
        policy=GaussianMLPModelCfg(
            hiddens=[256, 128, 64],
        ),
        value=DeterministicMLPModelCfg(
            hiddens=[256, 128, 64],
        ),
    )

    agent: AgentCfg = PPOAgentCfg(learning_rate=1e-4)

    experiment: ExperimentCfg = ExperimentCfg(
        directory="humanoid", 
        experiment_name="example", 
        write_interval=128, 
        checkpoint_interval=5000
    )

    trainer: TrainerCfg = TrainerCfg(timesteps=160000)

This config is later processed in skrl/train.py

Motivation

Architecture of ManagerBasedRLEnvCfg has modular architecture using @configclass, this is very neat design as it enables flexible manipulation with submodules, what we are benefiting from when running experiments across multiple RL tasks.

However RL config (for skrl lib) exists as a single yaml file. Unifying it to modular architecture identical to ManagerBasedRLEnvCfg would make experiment in Isaac-lab smoother, cleaner and esier.

In the future all RL libs could be using this type of configuration. It can insentivise them to reuse the modules across libraries. Eventually it could lead to desgin where all libs start from single train script and change your task to other library can be done just by toggling the rl_library parameter.

In addition this architecture allows easy extention of current functionality by external libraries:

(e.g. I am developing a library for model instantiation, that should be modular and powerful enough to instantiate any model's architecture easily. If the Isaac lab RL configuration is modular, one can simply switch the model config for this new lib model config.)

Alternatives

Proposed change could live directly in ManagerBasedRLEnvCfg. (however this only makes sense in the future, if this was wildly adopted by all other RL libs in Isaac-lab)

Actionables

Please let me know if this contribution is welcomed (and not conflicting with some of your plans) and I will issue a PR for it (already have a working version in my company Isaac-lab mirror, just need to polish it and modify for use in dev isaac-lab).