Where "METAFLOW_DATATOOLS_SYSROOT_S3" config is used?

[zimmy-upstage]

When configuring aws variables with metaflow configure aws, we set METAFLOW_DATATOOLS_SYSROOT_S3. But I couldn't find where this config is used. It's only used in cli. I guess this line should be changed to METAFLOW_DATATOOLS_SYSROOT_S3 from METAFLOW_DATATOOLS_S3ROOT.

[romain-intel]

So, the code you refer to is this:

DATATOOLS_S3ROOT = from_conf(
    "METAFLOW_DATATOOLS_S3ROOT",
    os.path.join(from_conf("METAFLOW_DATASTORE_SYSROOT_S3"), DATATOOLS_SUFFIX)
    if from_conf("METAFLOW_DATASTORE_SYSROOT_S3")
    else None,
)

and it does seem to be conflicting with https://github.com/Netflix/metaflow/blob/master/metaflow/main_cli.py#L490 which sets the wrong environment variable. For consistency, we should probably use METAFLOW_DATATOOLS_SYSROOT_S3 as you suggest. Feel free to make a PR to make this fix.

@oavdeev for viz in case this value is used elsewhere. Currently there are two so we should pick one knowing that it may break things.

[savingoyal]

This indeed looks like a bug that should be fixed.

[roofurmston]

@savingoyal @oavdeev Is this bug still open. From the configuration generated I would assume so.

Happy to open an PR, if so.

[tfurmston]

I've made a PR for this issue here - https://github.com/Netflix/metaflow/pull/1312