[BUG] Setting experiment id from python fails when running script from CLI

[cyruoff]

System information

• OS Platform and Distribution: Linux Ubuntu 18.04
• MLflow installed from: binary
• MLflow version: 1.8
• Python version: 3.8

Describe the problem

Setting the experiment id in a python script with mlflow.set_experiment('foo') does not work when the script is run with mlflow run . -e train_model_min. In the case where something is logged to the experiment, the run fails with the exception

mlflow.exceptions.MlflowException: Cannot start run with ID c21d2c4e75c047608003235f213e5bb5 because active run ID does not match environment run ID. Make sure --experiment-name or --experiment-id matches experiment set with set_experiment(), or just use command-line arguments

even though no experiment id was specified in the CLI.

If nothing is logged (commenting out the line with log_metric in the code below), the run is executed without error but is logged under the Default experiment instead of foo.

Code to reproduce issue

train_model_min.py:

import mlflow

def main():
    mlflow.set_experiment('foo')
    mlflow.log_metric('answer', 42)

if __name__ == '__main__':
    main()

MLproject file:

name: MLFlow Test

conda_env: conda.yaml

entry_points:
  train_model_min:
    command: "python train_model_min.py"

command to reproduce bug: mlflow run . -e train_model_min

[cthacker]

@cyruoff following up on this, I noticed that it will not set the run_name either.

To fix your issue you have to pass in the experiment name in the command line like so: mlflow run --experiment-name foo . Though I don't think this should be expected behavior.

That will at least properly set everything under that experiment. However a problem I'm having is that I cannot set the run_name via with mlflow.start_run(run_name='myrun') as run:

[cyruoff]

@cthacker Thanks for your follow-up. You are right, setting the run id with mlflow.start_run when running with the CLI mlflow run ... does not work either. (I know that I can set the experiment with a command line argument, but this should not be the only way, since it is very impractical) Maybe you should file another bug report and link to this one?

Both features, setting the run id and experiment id from within python, work when you simply start your script with python without the CLI, the bug is connected to mlflow run ...

[kivancic-ut]

Hi, I'm still having this issue. I'm running one of the example scripts, mlflow/examples/sklearn_elasticnet_wine.

If I position myself into the examples folder and run: mlflow run sklearn_elasticnet_wine, the run gets logged in the Default experiment in the UI.

If I set the experiment name in the CLI, like this: mlflow run sklearn_elasticnet_wine --experiment-name tutorial, it goes into the tutorial experiment. So far so good.

But, if I try adding this line to mlflow/examples/sklearn_elasticnet_wine/train.py: mlflow.set_experiment("tutorial"), just above line 58, with mlflow.start_run():, and run mlflow run sklearn_elasticnet_wine, I get the following error:

2021/03/26 12:41:14 INFO mlflow.projects.utils: === Created directory /var/folders/9b/f_tsf7sd0_96pv3l8b6j8lpm0000gn/T/tmp5w27jybm for downloading remote URIs passed to arguments of type 'path' ===
2021/03/26 12:41:14 INFO mlflow.projects.backend.local: === Running command 'source /Users/kivancic/miniconda3/bin/../etc/profile.d/conda.sh && conda activate mlflow-6284a367a61b51ccdf445333a216776597fb4efc 1>&2 && python train.py 0.5 0.1' in run with ID '4f043b57aadd4005b4c0cc5e3df92547' ===
INFO: 'tutorial' does not exist. Creating a new experiment
Traceback (most recent call last):
  File "train.py", line 60, in <module>
    with mlflow.start_run():
  File "/Users/kivancic/miniconda3/envs/mlflow-6284a367a61b51ccdf445333a216776597fb4efc/lib/python3.6/site-packages/mlflow/tracking/fluent.py", line 193, in start_run
    "arguments".format(existing_run_id)
mlflow.exceptions.MlflowException: Cannot start run with ID 4f043b57aadd4005b4c0cc5e3df92547 because active run ID does not match environment run ID. Make sure --experiment-name or --experiment-id matches experiment set with set_experiment(), or just use command-line arguments
2021/03/26 12:41:16 ERROR mlflow.cli: === Run (ID '4f043b57aadd4005b4c0cc5e3df92547') failed ===

I don't want my whole team to remember to set a CLI flag every time they start a training run. I want us to commit that to GitHub and not think about it again.

Is there a way right now to do this programmatically?

[MadanMl]

I have the same issue with: Python version: 3.7 MLflow version: 1.14.1, Installed using pip This is so irritating

[Pooja1905]

I am running into the same issue. Any help on this is appreciated.

[tcmle]

same issue still

[nicolo-toscano]

Same problem here

[nasserboan]

Setting up a MLflow talk to some friends on PyData's chapter here in Brasília (Brazil)! Got the same problem, is there any updates on this? I was able to circumvent this calling the --experiment-name on my CLI, but it would be nice to make this programmatically.