[Bug]: When `scratch_dir` is set, the updated slurm stdout file is overwritten

[Andrew-S-Rosen]

Code snippet

import shlex

from custodian.custodian import Custodian
from custodian.vasp.jobs import VaspJob
from custodian.vasp.handlers import VaspErrorHandler

scratch_dir = "/scratch/gpfs/ROSENGROUP/asrosen/custodian_scratch"
full_vasp_cmd = "srun --nodes 1 --ntasks-per-node 112 --exclusive vasp_std"
full_vasp_gamma_cmd = "srun --nodes 1 --ntasks-per-node 112 --exclusive vasp_gam"

split_vasp_cmd = shlex.split(full_vasp_cmd)
split_vasp_gamma_cmd = shlex.split(full_vasp_gamma_cmd)

jobs = [VaspJob(split_vasp_cmd, gamma_vasp_cmd=split_vasp_gamma_cmd)]

c = Custodian(
    handlers=[VaspErrorHandler()],
    jobs=jobs,
    scratch_dir=scratch_dir,
)

c.run()

What happened?

When specifying scratch_dir in Custodian, it will automatically copy everything from the starting directory to the scratch directory. When the Custodian run is done, it will then copy everything out of the scratch directory back into the starting directory.

https://github.com/materialsproject/custodian/blob/0c1b6a562dd84bfeae66181bf25db72ca9b1c39e/src/custodian/custodian.py#L368-L372

If you launch a Slurm job in /starting/path, it will create /starting/path/slurm-12345.out and /starting/path/scratch_link that links to scratch_dir. It will copy slurm-12345.out to the scratch_dir, but only the original /starting/path/slurm-12345.out is updated. When the calculation finishes, the /starting/path/slurm-12345.out file is overwritten with the copied file from the start, destroying any logging.

Version

2025.5.12

Which OS?

• [ ] MacOS
• [ ] Windows
• [ ] Linux

Log output