cylc-flow icon indicating copy to clipboard operation
cylc-flow copied to clipboard

symlink dirs: support symlinking log/job

Open oliver-sanders opened this issue 1 year ago • 2 comments

We support symlinking the log dir but we don't support linking log/job differently to log.

The function of files in the log dir differs. The workflow log files are highly important live files. The job log dirs are relatively dispensable.

As such it may make sense to symlink the log/job files onto a larger, but less reliable filesystem and keep the workflow logs & public DB on a smaller but more dependable FS.

https://github.com/cylc/cylc-flow/blob/dfd316a4fea83b14c64c05f4ed4d9b5fd2be8f99/cylc/flow/cfgspec/globalcfg.py#L1151-L1162

Make sure to check cylc clean is correctly set up to remove this directory if symlinked.

oliver-sanders avatar Jun 17 '24 08:06 oliver-sanders

Why not just go generic and granular:

        with Conf('symlink dirs',  # noqa: SIM117 (keep same format)
                  desc="""
        ...
        """):
            with Conf('<install target>', desc="""
                :ref:`Host <Install targets>` on which to create the symlinks.
            """):
                Conf('<target>', VDR.V_STRING, None, desc="""
                    Alternative location for the <target> dir.

wxtim avatar Jun 17 '24 09:06 wxtim

Because there isn't an obvious use case for most possible combinations and that would not be supported by cylc clean and may clash with Cylc's expected lifecycle. The implementation should be generic though.

oliver-sanders avatar Jun 17 '24 09:06 oliver-sanders