Improved loading of jupyterhub_config.d folder configs

[consideRatio]

I was looking to emit log messages when loading the config, but I note that that were other things that differed the z2jh way of loading additional configs from the tljh way of loading additional configs.

I'd love help to extract what makes sense with regards to this, and do it in both z2jh and tljh.

Current way of doing it

TLJH

# Load arbitrary .py config files if they exist.
# This is our escape hatch
extra_configs = sorted(glob(os.path.join(CONFIG_DIR, 'jupyterhub_config.d', '*.py')))
for ec in extra_configs:
    load_subconfig(ec)

Z2JH

# load /usr/local/etc/jupyterhub/jupyterhub_config.d config files
config_dir = "/usr/local/etc/jupyterhub/jupyterhub_config.d"
if os.path.isdir(config_dir):
    for file_path in sorted(glob.glob(f"{config_dir}/*.py")):
        file_name = os.path.basename(file_path)
        print(f"Loading {config_dir} config: {file_name}")
        with open(file_path) as f:
            file_content = f.read()
        # compiling makes debugging easier: https://stackoverflow.com/a/437857
        exec(compile(source=file_content, filename=file_name, mode="exec"))

Notable differences to remove

• [ ] We don't print information that we are loading the config in tljh We should print in TLJH also.
• [ ] We use load_subconfig in tljh, and exec + compile in z2jh Q: Should we go for load_subconfig in z2jh, or exec + compile in tljh, or have it remain different? A: ?

@minrk can you comment on what approach you think we should use?

[minrk]

We should print in TLJH also.

👍

Should we go for load_subconfig in z2jh

I don't see a reason to not use load_subconfig. I don't recall the reasoning behind the exec/compile.

There is one practical difference: exec/compile is effectively concatenating jupyterhub_config.py (meaning all existing variables are available), whereas load_subconfig is more akin to import - it's an isolated module which produces a config object, which is then merged.

• Pro exec/compile: it lets you build one config file with snippets that may rely on utilities such as set_config_if_not_none. It also lets you reference previously configured values (e.g. c.JupyterHub.whatever)
• Con exec/compile: all the state available by name in jupyterhub_config.py become de facto public APIs due to their availability, rather than treating each file as independent.

There's a very good chance that exec(compile is used because folks were not aware of load_subconfig. My inclination would be to use load_subconfig as it's traitlets.config's official API for doing exactly what we use it for unless there's a specific intended case that requires stitching-together a single namespace. Probably worth checking the history for whether exec was switched to vs just how it's always been.