Run args not correctly logged in hyperparameters view

[ivan94fi]

Describe the Bug

The command line arguments passed to the python script are sometimes not correctly logged as run_arg_<number>, but are just logged as hyperparameters. In my tests, adding a long option causes this bug, but this might be coincidental.

The experiment constructor is called like this:

experiment = Experiment(
    project_name="...",
    auto_metric_logging=False,
    log_graph=False,
    disabled=config.no_comet,
    auto_output_logging="simple",
    log_git_patch=False,
    display_summary_level=0,
)

and the parameters are logged like so:

config: argparse.Namespace = get_config()  # parse the CLI arguments
experiment.log_parameters(vars(config))

The cli arguments of interest are: -w 12 train -e 80 --pixel-shuffle, and are translated in the following hyperparameters, without any "run arg" (please ignore any other hyperparameter, they all have default values which are not changed in these runs):

Screenshot 1:

Screenshot 2:

Screenshot 3:

As can be seen in these images, all the mentioned options, workers, pixel-shuffle and epochs are repeated (e is the short version of epochs, w is the same for workers), as both the cli option and the actual hyperparameter in the parsed argparse.Namespace are logged . Actually the pixel-shuffle option ends up picking a wrong value, which should be assigned to the command option (not showed in screenshots).

Expected behavior

All the run args should be logged, together with the corresponding parameter: the arguments -w 12 train -e 80, that is, the same as before but without the pixel-shuffle long option, are correctly logged, and run args appear as expected:

From these tests it seems that adding the --pixel-shuffle option actually causes the bug, but, again, this does not appear as a sound conclusion, just an observation.

Where is the issue?

• [x] Comet Python SDK
• [x] Comet UI
• [ ] Third Party Integrations (Huggingface, TensorboardX, Pytorch Lighting etc)

This issue seems to appear with the last version of the comet web UI, but the older experiments still show the run args correctly in the new UI, so this may be a coincidence and maybe it is a bug in the python SDK.

The issues appear both with python SDK version 3.14.1 and the latest version 3.23.0 (I tested only these two versions).

To Reproduce

[EDIT] With this script I am able to reproduce the bug independently from my project:

• python test.py -a a train -e e: correct hyperparameters logged.
• python test.py -a a --bool-argument train -e e: wrong hyperparameters logged

So after all maybe it is in fact the boolean argument (declared with action="store_true") that causes the problem.

test.py:

from comet_ml import Experiment
from argparse import ArgumentParser

parser = ArgumentParser()

parser.add_argument("-a", "--argument-a")
parser.add_argument("-b", "--argument-b")
parser.add_argument("-c", "--argument-c")
parser.add_argument("--bool-argument", action="store_true")

subparsers = parser.add_subparsers(
    title="Subparsers",
    description="Description",
    dest="command",
    required=True,
)
train_parser = subparsers.add_parser("train")
train_parser.add_argument("-d", "--argument-d")
train_parser.add_argument("-e", "--argument-e")
train_parser.add_argument("-f", "--argument-f")

config = parser.parse_args()

experiment = Experiment(
    project_name="test",
    auto_metric_logging=False,
    log_graph=False,
    auto_output_logging="simple",
    log_git_patch=False,
    display_summary_level=0,
)

experiment.log_parameters(vars(config))

[DN6]

Thanks for raising this @ivan94fi and providing so much context. Let me take a look at what's going on here. I will get back to you as soon as possible.

[ivan94fi]

Thank you @DN6 for the quick reply. I updated the bug report with a script to reproduce the bug. Hope it helps.

[DN6]

Hello @ivan94fi . I was able to reproduce your issue. I believe boolean flags and subparsers can cause issues with our autologger's command line parser. Our SDK team is working on a solution to address these edge cases.

Another thing I noticed is that you are logging the parsed args with experiment.log_parameter but you have not disabled the autologger's command line parser. Can you update your Experiment object with the following configuration and try rerunning your script to see if it produces the desired output

experiment = Experiment(
    project_name="test",
    auto_metric_logging=False,
    log_graph=False,
    auto_output_logging="simple",
    log_git_patch=False,
    display_summary_level=0,
    parse_args=False,
)

[ivan94fi]

Hi, adding parse_args=False avoids the problem, the log_parameters call does its job and the config dictionary is correctly logged, but as expected the run args are not logged, and my intention is to have them logged.

By leaving parse_args=True, and removing the log_parameters call, I only get the options passed to the command line, while I want to log all the options with their values.

So there seems to be an interference between the log_parameters, which I believe is used to log whatever value as a parameter, and the parse_args argument to Experiment constructor, which should only report arguments from the command line.

For now I am logging the cli arguments manually, as in the snippet below, until you guys solve the issue. Thank you for your time.

experiment = Experiment(
    project_name="test",
    auto_metric_logging=False,
    log_graph=False,
    auto_output_logging="simple",
    log_git_patch=False,
    display_summary_level=0,
    parse_args=False,
)
 
experiment.log_parameters(vars(config) | {"cli_args": " ".join(sys.argv[1:])})

[DN6]

Hi @ivan94fi. Just so I understand your use case correctly. You would like to log the parsed args, and also log the run args as separate parameters in order to track which args were modified at runtime?

The parsed args logged as parameters should reflect the values passed in at runtime? I just want to make sure I understand the expected behavior correctly here.

[ivan94fi]

Hi @DN6.

You would like to log the parsed args, and also log the run args as separate parameters in order to track which args were modified at runtime?

The parsed args logged as parameters should reflect the values passed in at runtime?

Yes, these are correct. As a matter of fact I just want the arguments string (sys.argv) to be logged somewhere, and as Comet itself logged that in the Hyperparameters section, I just sticked with this convention, which seems reasonable.

The rationale behind this use case is what you mentioned: I want to be able to see all the hyperparameters, and also which parameter is changed at runtime when executing the experiment.

[github-actions[bot]]

This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 5 days.

[github-actions[bot]]

This issue was closed because it has been stalled for 5 days with no activity.