Running frameworks from a forked repo

[Innixma]

Given the current functionality:

python runbenchmark.py {framework}:latest

which is equivalent to building from source from the master branch,

I'd like to be able to do:

# Using the primary repo
python runbenchmark.py {framework}:{branch}

# Using a forked repo
python runbenchmark.py {framework}:{user}/{branch}

For example, say I wanted to test this branch from a forked repo of AutoGluon.

Ideally, I'd like it to work via the following command:

python runbenchmark.py AutoGluon:gradientsky/20201215_fastai_nn_updates

This seems do-able by some simple regex checking for : and / characters in the input. If no {user} was found, then it can default to the main user for the framework (awslabs in the case of AutoGluon).

I think this functionality would be very valuable as it would allow contributors of the various frameworks to use automlbenchmark as a way to validate their changes prior to sending a PR / merging.

While I believe I could implement this for AutoGluon, I'm making this issue in case there is a generic solution that would be better applied to all frameworks that is easier to maintain.

[PGijsbers]

Good suggestion 👍

We don't have the time to work on this right away. It seems the installation logic is duplicated for most Python frameworks:

if [[ "$VERSION" == "stable" ]]; then
    PIP install --no-cache-dir -U ${PKG}
elif [[ "$VERSION" =~ ^[0-9] ]]; then
    PIP install --no-cache-dir -U ${PKG}==${VERSION}
else
#    PIP install --no-cache-dir -e git+${REPO}@${VERSION}#egg={PKG}
    TARGET_DIR="${HERE}/lib/${PKG}"
    rm -Rf ${TARGET_DIR}
    git clone --depth 1 --single-branch --branch ${VERSION} --recurse-submodules ${REPO} ${TARGET_DIR}
    PIP install -U -e ${TARGET_DIR}
fi

I think a more generic solution is to extract this to a separate shell script in frameworks/shared which can be called by the individual framework setup. But it would also require updates to interpretation of the command line argument, and an update to how we log the framework versions.

[sebhrusen]

Hi @Innixma, technically, you can already run benchmarks against any branch, and even on fork repo, but not directly from the command line. The recommended approach is to add a framework definition for each version in your custom frameworks.yaml (see https://github.com/openml/automlbenchmark/blob/master/docs/HOWTO.md#framework-definition). For example:

AutoGluon_myrepo:
   extends: AutoGluon
   setup_args: ['my_branch', 'url_to_my_repo']  # list of `*args` passed to `setup` in  `AutoGluon/__init__.py`
                                                # their semantic is defined in AutoGluon/setup.sh

AutoGluon_mybranch:
   extends: AutoGluon
   version: 'my_branch'      # of course, also possible to use setup_args as above

if testing locally, better run the command with --setup force to override the local version.

I'm not personally convinced that the command line syntactic sugar is needed: it will mainly add complexity, just to support a rarely used feature that is already supported in a more standard way.

[Innixma]

Hi @sebhrusen , the approach you mention is what I am currently doing, however I'd like to make the case for how it could be improved:

When using AWS mode, each EC2 instance clones a specific fork/branch of automlbenchmark. This means that any time I want to test a new branch of a framework, I need to commit my framework definition code change and push it so it is available to be git cloned by the EC2 instances.

While someone familiar with automlbenchmark will be able to know which files to update to correctly launch the AWS mode using a personal fork of automlbenchmark, it is very confusing for someone new to automlbenchmark. This confusion also exists for local mode runs too, as they would need to still make the code edits in automlbenchmark to run their branch.

We'd ideally want users who make changes to AutoGluon to include a quick local test run of automlbenchmark in their contribution PR. To encourage contributors, we'd want this process to be as simple as possible (ideally no code edits to automlbenchmark required).

Hope this clarifies the benefits of the functionality proposed, and thanks again for all the hard work that goes into automlbenchmark, its really awesome!

[sebhrusen]

@Innixma When testing a different version of AG, you shouldn't need to do any change to the automlbenchmark application itself, except when it requires a change in the integration code (exec.py and so on).

The documentation I was pointing to in my previous message is probably not clear, so let me try to explain in detail.

In the top of https://github.com/openml/automlbenchmark/blob/master/resources/config.yaml you can read:

project_repository: https://github.com/openml/automlbenchmark#stable      # this is also the url used to clone the repository on ec2 instances
                                                                          # when running those without docker.
                                                                          # to clone a specific branch/tag, add a url fragment, e.g.:
                                                                          # https://github.com/openml/automlbenchmark#stable

user_dir: ~/.config/automlbenchmark   # where to override settings with a custom config.yaml file and, for example, add custom frameworks, benchmark definitions or framework modules.
input_dir: ~/.openml/cache            # where the datasets are loaded by default.
output_dir: results                   # where logs and results are saved by default.

root_dir:   # app root dir: set by caller (runbenchmark.py)

these are the most important properties when you want to extend/customize the app.

• user_dir is the folder where you'll be able to create your own config.yaml and plug custom frameworks or benchmark definitions (more about this later, because that's what we'll need to change for you to test different versions of AG without modifying the automlbenchmark): to explain quickly, we can list there some files that will get automatically uploaded to ec2 instances, without the need to submit them to the repo.
• input_dir is the folder containing the datasets. By default it points to the openml cache folder. It's also the folder where the non-openml datasets are downloaded (see https://github.com/openml/automlbenchmark/blob/master/docs/HOWTO.md#file-datasets).
• output_dir is the folder where the results are created or downloaded.
• root_dir doesn't usually change, but I'm mentioning it here for completeness: it is just pointing to the root directory of the automlbenchmark application itself, and is mainly used to be able to call the app from outside this root directory.

Note that all those folders (except the root) can also be defined on the command line:

python runbenchmark.py --help
…
  -i input_dir, --indir input_dir
                        Folder where datasets are loaded by default. Defaults
                        to `input_dir` as defined in resources/config.yaml
  -o output_dir, --outdir output_dir
                        Folder where all the outputs should be written.
                        Defaults to `output_dir` as defined in
                        resources/config.yaml
  -u user_dir, --userdir user_dir
                        Folder where all the customizations are stored.
                        Defaults to `user_dir` as defined in
                        resources/config.yaml

Now the most important for your customization is user_dir. Each time you run python runbenchmark.py …, the app will search for a config.yaml in this user_dir and merge it on top of the default resources/config.yaml. Therefore, if you have a ~/.config/automlbenchmark/config.yaml file, it will always be applied except if you change the user_dir on the command line to point it somewhere else (note that this also allows you to switch easily between various configurations).

Now that you have this ~/.config/automlbenchmark/config.yaml, the app becomes quite flexible. In particular, you can now plug additional framework definitions files, benchmark definition files… you can even plug additional frameworks that are not in the repo (see https://github.com/openml/automlbenchmark/tree/master/examples/custom for an example showing how to do this).

Let's look at the config.yaml in this example:

project_repository: https://github.com/openml/automlbenchmark

input_dir: '{user}/data'   # change the default input directory (where data files are loaded and/or downloaded).

frameworks:
  definition_file:  # this allows to add custom framework definitions (in {user}/frameworks.yaml) on top of the default ones.
    - '{root}/resources/frameworks.yaml'
    - '{user}/frameworks.yaml'

benchmarks:
  definition_dir:  # this allows to add custom benchmark definitions (under {user}/benchmarks) to the default ones.
    - '{user}/benchmarks'
    - '{root}/resources/benchmarks'
  constraints_file: # this allows to add custom constraint definitions (in {user}/constraints.yaml) on top of the default ones.
    - '{root}/resources/constraints.yaml'
    - '{user}/constraints.yaml'

aws:
  resource_files:  # this allows to automatically upload custom config + frameworks to the running instance (benchmark files are always uploaded).
    - '{user}/config.yaml'
    - '{user}/frameworks.yaml'
    - '{user}/constraints.yaml'
    - '{user}/benchmarks'
    - '{user}/extensions'

• project_repository just says that ec2 instances will still use the default repo, but the master branch (no url fragment pointing to a specific branch).
• frameworks adds a local {user}/frameworks.yaml definition to the default one.
• same for benchmarks and constraints.
• more importantly, aws.resource_files lists all the files and directories that will be uploaded to the ec2 instances (through s3). There's also a filter applied to upload only relevant files (see resource_ignore in the default config.yaml, of course this can also be overridden in the custom config.yaml).

In this file you can also notice some placeholders like {root}, {user} (but we can also use {input} and {output}) that map to their xxx_dir counterpart that we described earlier. Those placeholders are important because they allow those files to be used both locally and on the ec2 instances. If we use absolute paths in this config file, it's very likely that it won't work on ec2, so it is recommended to put all custom definitions, extensions and so on, relatively to the {user} folder to ensure they will get copied to the right place and loaded correctly by the benchmark app running on ec2.

Now we have almost all the pieces of the puzzle:

• you can use this yaml file as the base for your own {user}/config.yaml (by default ~/.config/automlbenchmark/config.yaml).
• you can define your custom versions of AG in {user}/frameworks.yaml (i.e. ~/.config/automlbenchmark/frameworks.yaml):

AutoGluon_myrepo:
   extends: AutoGluon
   setup_args: ['my_branch', 'url_to_my_repo']  # list of `*args` passed to `setup` in  `AutoGluon/__init__.py`
                                                # their semantic is defined in AutoGluon/setup.sh

AutoGluon_mybranch:
   extends: AutoGluon
   version: 'my_branch'      # of course, also possible to use setup_args as above

• you can also define local datasets if you need, for example in {user}/benchmarks/s3datasets.yaml (see https://github.com/openml/automlbenchmark/blob/master/docs/HOWTO.md#file-datasets for the syntax).
• you can define your own constraints (larger instances, shorter/longer durations...).
• you can run the whole the same way as any other benchmark, for example:

> python runbenchmark.py AutoGluon_myrepo -m aws
> python runbenchmark.py AutoGluon_myrepo s3datasets -m aws

for example, you could have an .amlb_config folder in AG repo with a config.yaml and so on, add a git hook (post-commit? pre-push?) that would update the .amlb_config/frameworks.yaml (adding a definition or updating the branch in AutoGluon_mybranch), and a run.sh in the same folder that triggers the benchmark with the right params (especially, ensuring that the correct config is used using the -u param):

python /path/to/runbenchmark.py AutoGluon_local s3datasets -u .

so that contributors would just have to run:

.amlb_config/run.sh

If this lengthy message didn't answer your question, then let me apologize, I probably didn't understand your requirement.

[Innixma]

@sebhrusen Thanks for the incredibly detailed explanation!

I think the main piece I had been missing was the AWS config options to ensure the local configs/frameworks are copied to the AWS instances. I had tried this a long time ago and couldn't get it to work because I wasn't using the AWS options. I also wasn't aware of the {user} etc. options, which are very nice.

I believe what you described covers my use-case. Over the next couple weeks I'll be looking into this more and trying to get it working with custom datasets, I'll let you know if I run into any issues.

I think the original functionality request is still valid due to its simplicity (since it would automatically work for most frameworks without the framework authors having to do the steps mentioned), however this mostly applies to local runs as AWS runs will still require additional configuration such as AWS account, buckets, etc. which lean more towards the solution you described.

[eddiebergman]

I would also think a command line feature would be useful to avoid having setup scripts to reconfigure framework files each time. My use case is to run regression tests against PR's contributed to autosklearn which often come from forks. We would like to keep track of tests against specific branches and commit hashes which are not always coming from the main automl/autosklearn repo. These are set up and torn down so having to create a framework file and add in the relevant tag each time is not intuitive for this kind of use case.

In general, having command line arguments to run one shot framework definitions would be very helpful. This could be made as elaborated as needed but for now one that simply extends of the already present frameworks would probably be the most easy. A suggested version which would require less regex parsing would be:

python runbenchmark.py my_framework_def test \ 
    --extends autosklearn \
    --version branch \ 
    --setup_args "branch 'https://my-git-enabled-repo'" \
    --other-framework-def-param value

The suggested version at the top might run into parsing conflicts when searching for framework definition files as indicated by the :.

My current setup is to simply modify the tags in config.yml by appending some word so

# config.yml
    tags: [..., fork_branch]

And then add a custom frameworks_fork_branch.yml file

# frameworks_fork_branch.yml
autosklearn:
    version: "branch"
    repo: "git@my_repo.com/fork.git"

Also for the example you provided above for setup_args, it doesn't work with a list as it gets overwrited with framework.version and framework.repo if it exists The relevant code is in automlbenchmark/amlb/frameworks/definitions.py:105

def _add_default_setup_args(framework):
   # "long,string" gets converted to []" 
    if "setup_args" in framework and isinstance(framework.setup_args, str):
        framework.setup_args = [framework.setup_args]
    else:
        framework.setup_args = [framework.version]
        if "repo" in framework:
            framework.setup_args.append(framework.repo)