`SameFileError` when running sequential VASP jobs in the same directory

[Andrew-S-Rosen]

Describe the bug When running a VASP Flow locally without creating new directories, I get back a SameFileError. This is not particularly surprising, but I am wondering if there is a better way for atomate2 to handle this. Perhaps we should just be overwriting the files that are there, like what would happen in a normal VASP job submitted without any workflow orchestration tool? The reason this came up was because I was troubleshooting by running jobs locally and did not bother to rely on create_folders=True when launching the calculations. This is with Atomate2 0.0.21.

from atomate2.vasp.jobs.matpes import MatPesGGAStaticMaker
from pymatgen.io.vasp.sets import MatPESStaticSet
from pymatgen.core import Structure
from jobflow import Flow
from jobflow import run_locally

structure = Structure.from_file("test.cif")

toy_input_set = MatPESStaticSet(user_incar_settings={"ENCUT": 400, "KSPACING": 0.5})
static1 = MatPesGGAStaticMaker(input_set_generator=toy_input_set).make(structure)

static2 = MatPesGGAStaticMaker(name="Test 2", input_set_generator=toy_input_set, copy_vasp_kwargs={"additional_vasp_files": ("WAVECAR",)}).make(static1.output.structure, prev_dir=static1.output.dir_name)

flow = Flow([static1, static2])

run_locally(flow)

Traceback (most recent call last):
  File "/u/rosen1/.local/lib/python3.11/site-packages/jobflow/managers/local.py", line 117, in _run_job
    response = job.run(store=store)
               ^^^^^^^^^^^^^^^^^^^^
  File "/u/rosen1/.local/lib/python3.11/site-packages/jobflow/core/job.py", line 604, in run
    response = function(*self.function_args, **self.function_kwargs)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/u/rosen1/.local/lib/python3.11/site-packages/atomate2/vasp/jobs/base.py", line 214, in make
    copy_vasp_outputs(prev_dir, **self.copy_vasp_kwargs)
  File "/u/rosen1/.local/lib/python3.11/site-packages/atomate2/utils/file_client.py", line 586, in gen_file_client
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/u/rosen1/.local/lib/python3.11/site-packages/atomate2/vasp/files.py", line 89, in copy_vasp_outputs
    copy_files(
  File "/u/rosen1/.local/lib/python3.11/site-packages/atomate2/utils/file_client.py", line 588, in gen_file_client
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/u/rosen1/.local/lib/python3.11/site-packages/atomate2/common/files.py", line 73, in copy_files
    file_client.copy(from_file, to_file, src_host=src_host)
  File "/u/rosen1/.local/lib/python3.11/site-packages/atomate2/utils/file_client.py", line 240, in copy
    shutil.copy2(src_filename, dest_filename)
  File "/u/rosen1/.conda/envs/cms/lib/python3.11/shutil.py", line 448, in copy2
    copyfile(src, dst, follow_symlinks=follow_symlinks)
  File "/u/rosen1/.conda/envs/cms/lib/python3.11/shutil.py", line 236, in copyfile
    raise SameFileError("{!r} and {!r} are the same file".format(src, dst))
shutil.SameFileError: PosixPath('/work/nvme/bems/rosen1/test/test/INCAR.gz') and PosixPath('/work/nvme/bems/rosen1/test/test/INCAR.gz') are the same file

Expected behavior A clear and concise description of what you expected to happen.

Screenshots If applicable, add screenshots to help explain your problem.

[esoteric-ephemera]

Sorry for getting back slowly on this - happy to discuss possibilities for what an accepted behavior is. The safest would be something like gzipping the files that are already there and raising a warning to the user

[JaGeo]

I believe this might be something to solve on the jobflow level, right?

[esoteric-ephemera]

Not sure where the best place is for a fix - shutil.copy is being called in atomate2's file client, but I'd imagine that run_locally(flow, create_folders=False) could be problematic for similar reasons in more flows. @Andrew-S-Rosen do you want to link an issue on jobflow and see what the response is?

[Andrew-S-Rosen]

I am also unsure where the best solution for this is.

I am imagining a scenario where the user will run a relax --> static calculation. They launch the calculation with Atomate2 and Jobflow locally. If the files are moved or renamed by Jobflow, that will cause a problem because the information about where the set of files are will be broken. If it's gzipped, it may be okay from Atomate2's perspective.

Personally, I still think simply copying the files regardless of whether it exists or not is the simplest solution. That said, this is of course not some huge issue --- the user can always use create_folders=True in practice.

[JaGeo]

@Andrew-S-Rosen While it is not a big deal, I also ean into this issue once and was confused for 30 sec. I think it would be good to have a general solution