`ensure_path_nonexistent` doesn't remove symlink

[zagy]

  File "/srv/s-oldb/deployment/.batou/00086ca4/lib/python3.8/site-packages/batou/component.py", line 320, in deploy
    sub_component.deploy(predict_only)
  File "/srv/s-oldb/deployment/.batou/00086ca4/lib/python3.8/site-packages/batou/component.py", line 320, in deploy
    sub_component.deploy(predict_only)
  File "/srv/s-oldb/deployment/.batou/00086ca4/lib/python3.8/site-packages/batou/component.py", line 339, in deploy
    self.update()
  File "/srv/s-oldb/deployment/.batou/00086ca4/lib/python3.8/site-packages/batou/lib/file.py", line 616, in update
    ensure_path_nonexistent(self.target)
  File "/srv/s-oldb/deployment/.batou/00086ca4/lib/python3.8/site-packages/batou/lib/file.py", line 23, in ensure_path_nonexistent
    shutil.rmtree(path)
  File "/run/current-system/sw/lib/python3.8/shutil.py", line 728, in rmtree
    onerror(os.path.islink, path, sys.exc_info())
  File "/run/current-system/sw/lib/python3.8/shutil.py", line 726, in rmtree
    raise OSError("Cannot call rmtree on a symbolic link")
OSError: Cannot call rmtree on a symbolic link

[zagy]

This could be a race condition due to parallel job execution and nfs.

[elikoga]

https://github.com/flyingcircusio/batou/blob/e6dabac30733f5a90205b9eb994e34aab21493a0/src/batou/lib/file.py#L21-L29

This has to be a race condition where 24 is evaluated as false but somewhere during execution of 26 and 27, os.path.islink(path) turns true

Not sure what to do about this. I think this is a valid error and I can't think of anything batou can do in that situation right now

[elikoga]

What about an unconditional os.rename to a random name at the beginning? If it fails due to src not existing, we are happy. If it does succeed we can remove the renamed file or directory.

If successful, the renaming will be an atomic operation (this is a POSIX requirement). from https://docs.python.org/3/library/os.html#os.rename