run_model_on_task: make avoid_duplicate_runs=False the default

[joaquinvanschoren]

Description

run_model_on_task has an option to avoid running experiments that already exist on OpenML, called avoid_duplicate_runs. This, however, requires an API key. It is currently the default, meaning that people can't try out this function without setting their API key. This creates an unnecessary obstacle, especially for beginners who don't know that the avoid_duplicate_runs option can be switched off.

Steps/Code to Reproduce

from sklearn import ensemble
from openml import tasks, runs

clf = ensemble.RandomForestClassifier()
task = tasks.get_task(3954)
run = runs.run_model_on_task(clf, task)

Expected Results

The model should just run. The user may have no intention to upload the run to OpenML later.

Actual Results

An API key error is thrown

Versions

Linux-5.4.188+-x86_64-with-Ubuntu-18.04-bionic Python 3.7.13 (default, Apr 24 2022, 01:04:09) [GCC 7.5.0] NumPy 1.21.6 SciPy 1.4.1 Scikit-Learn 1.0.2 OpenML 0.12.2

[PGijsbers]

I had a closer look, and the problem is actually a post request which should have been a get request: https://github.com/openml/openml-python/blob/99a62f609766db1d8a27ddc52cb619f920c052d0/openml/flows/functions.py#L256

With that fixed, you can run above code without an API key configured while still having the avoid_duplicate_runs functionality (at the cost of some additional server roundtrips to check if the run exists).

So now the question becomes, should we still prefer to have it turned off by default regardless?

[PGijsbers]

Also @mfeurer

[mfeurer]

That's an interesting question - maybe we can move this flag to the upload/publish function instead? It will serve the same purpose, but slightly improve the user experience as users can still run things without having to worry about duplicate runs.

[PGijsbers]

The idea of having it here is that the user may avoid unnecessary computation since they can identify there's a duplicate before running the experiment and download the results instead (not so much avoiding duplicates on the server, or so I thought).

[mfeurer]

Interesting, I thought it's to avoid duplicate stuff on the server. @joaquinvanschoren would you still like to remove that flag now that @PGijsbers has found a workaround?