Treat tags in python tasks the same as yaml tasks

[giuliolovisotto]

Following https://github.com/EleutherAI/lm-evaluation-harness/issues/2268 , I would like python tasks (ie, tasks defined by a class: !function my.class directive, to be treated the same as yaml tasks in terms of tags management.

Rationale

While I understand the shift from implementing tasks in python code to implementing them with yaml files, in my use-case writing python tasks subclassing ConfigurableTask makes for significantly less text overall. Unfortunately, at the moment, python tasks do not benefit from the tag management system offered by TaskManager. What would be really neat for my team is the possibility to use tags in python tasks, such as the following:

task: my-custom-task
class: !function source.MyClass
tag:
  - my-tag
...  # the rest of the config

currently this is not possible due to the different treatment of python tasks here:

                    if self._config_is_python_task(config):
                        # This is a python class config
                        tasks_and_groups[config["task"]] = {
                            "type": "python_task",
                            "yaml_path": yaml_path,
                        }
                    elif self._config_is_group(config):

and their exclusion from TaskManager._all_subtasks, (in fact they don't appear in lm-eval tasks --list)

Description of changes

• added tags defined in python tasks
• added python tasks to _all_subtasks
• added some basic tests for TaskManager that check correct tag handling. The test creates a temporary custom task that is passed as part of include_paths at the TaskManager init.

would close #2268

[CLAassistant]

All committers have signed the CLA.