How to configure variables in run_automlbenchmark.py?

[florence27]

Hi everyone,

thanks for this great repo, I am really looking forward to reproduce the benchmark results from your paper. I have a question though: I've been working my way through the README_automlbenchmark.txt. I executed the bash script and installed AutoGluon's custom automlbenchmark, that worked so far. Now, I want to execute the run_automlbenchmark.py, but I am a bit confused about how I should properly set the variables in the script. Maybe you could give me some help there? Would really appreciate that.

Thanks & Cheers!

[Innixma]

This means setting up your EC2 instance configurations in https://github.com/Innixma/autogluon-benchmarking/blob/master/autogluon_utils/setup/instance_config/config_defaults.py and setting up your AWS credentials so you can execute the EC2 fleet creation from the script.

Since it has been a fair amount of time since this benchmark was executed, it may be best for you to use a somewhat cleaner version produced by automlbenchmark as AutoGluon has now been added as a supported algorithm in it: https://github.com/openml/automlbenchmark

To run a test run locally:

git clone -b autogluon-workspace --single-branch https://github.com/Innixma/automlbenchmark.git

mkdir -p ~/virtual
python3 -m venv ~/virtual/automlbenchmark
source ~/virtual/automlbenchmark/bin/activate

cd automlbenchmark
pip install -r requirements.txt
cd ..

mkdir test_run
cd test_run

python ../automlbenchmark/runbenchmark.py AutoGluon_best test

If you want to do the first fold (of 10) of the full benchmark run done in the paper:

nohup python ../automlbenchmark/runbenchmark.py AutoGluon ag 1h8c -m aws -f 0 -p 50 > log_automlbenchmark_AutoGluon_1h8c.file 2>&1 &
nohup python ../automlbenchmark/runbenchmark.py AutoGluon ag 4h8c -m aws -f 0 -p 50 > log_automlbenchmark_AutoGluon_4h8c.file 2>&1 &
nohup python ../automlbenchmark/runbenchmark.py AutoGluon_best ag 1h8c -m aws -f 0 -p 50 > log_automlbenchmark_AutoGluon_best_1h8c.file 2>&1 &
nohup python ../automlbenchmark/runbenchmark.py AutoGluon_best ag 4h8c -m aws -f 0 -p 50 > log_automlbenchmark_AutoGluon_best_4h8c.file 2>&1 &

Be aware, this is 390 hours of m5.2xlarge compute and you will have to setup your AWS credentials as detailed in the automlbenchmark readme.

Best, Nick

[Innixma]

Here is a recent run output of AutoGluon with the latest v0.0.14 release done on October 1st 2020, in case you are interested. AutoGluon_automlbenchmark_2020_08_01_results.zip

AutoGluon_best is the configuration mentioned in our paper, and AutoGluon is the default settings not meant for high performance but ease of use. For comparisons with the paper, use AutoGluon_best.

[dev-rinchin]

Hi @Innixma!

I've checked AutoGluon_automlbenchmark_2020_08_01_results.zip, and there are results only for the 0th fold. Do you have results for all folds for AutoGluon_best? By the way, do you have results for the new 0.1.0 version?

Thanks!

[Innixma]

Hi @dev-rinchin,

Sorry for the late response, was on vacation when this was sent.

I've recently done a run on all 10 folds using the new benchmark suite detailed in https://github.com/openml/automlbenchmark/issues/187:

results_automlbenchmark_1h8c_2021_04_14_upgrade_cat.csv

This is for a pre-release of AutoGluon v0.1.1, but its pretty similar to v0.1.0.

Hope its useful!