Allow output_directory to be configured by the configuration file or CLI argument.

[baraksalomon]

Problem

Job outputs needs to be under a folder accessible by the UI. If the "jobs" folder is not under root_dir the download links are not created. I want the output folder not to be exposed in the UI but that download will still work.

Proposed solution

Allow to change the folder name from "jobs" to something like ".jobs" This will allow the files to be accessible by links by not be displayed in the UI. (hopefully)

Additional context

I tried configuring SchedulerApp.output_directory but the logs shows: [W 2024-12-23 09:49:44.196 SchedulerApp] Config option output_directory not recognized by SchedulerApp.

Exposing this parameter to the configuration will allow me to name it ".jobs" but its a generic solution for an existing parameter that may help others.

[andrii-i]

Thank you for creating this enhancement request @baraksalomon. We don't have resources to start working on this immediately but it does makes sense to implement.