cylc commands from a directory that does not exist should not cause failures (generally)

[ColemanTom]

Description

Running cylc commands from a directory that no longer exists causes failures. I suggest that whether the directory you are currently in exists or not should not matter (maybe there are some commands it does matter, but many should not, e.g. scan, tui).

Reproducible Example

$ d=$(mktemp -d)
$ cd "$d"
$ rmdir "$d"
$ cylc scan
shell-init: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory
Task exception was never retrieved
future: <Task finished name='Task-2' coro=<_AsyncPipe._generate() done, defined at /home/USER/miniconda3/envs/cylc-8.2.0/lib/python3.11/site-packages/cylc/flow/async_u
til.py:148> exception=FileNotFoundError(2, 'No such file or directory')>
Traceback (most recent call last):
  File "/home/USER/miniconda3/envs/cylc-8.2.0/lib/python3.11/site-packages/cylc/flow/async_util.py", line 151, in _generate
    async for item in gen.func(*gen.args, **gen.kwargs):
  File "/home/USER/miniconda3/envs/cylc-8.2.0/lib/python3.11/site-packages/cylc/flow/network/scan.py", line 222, in scan
    max_depth = glbl_cfg().get(['install', 'max depth'])
                ^^^^^^^^^^
  File "/home/USER/miniconda3/envs/cylc-8.2.0/lib/python3.11/site-packages/cylc/flow/cfgspec/glbl_cfg.py", line 22, in glbl_cfg
    return GlobalConfig.get_inst(cached=cached)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/USER/miniconda3/envs/cylc-8.2.0/lib/python3.11/site-packages/cylc/flow/cfgspec/globalcfg.py", line 1882, in get_inst
    cls._DEFAULT.load()
  File "/home/USER/miniconda3/envs/cylc-8.2.0/lib/python3.11/site-packages/cylc/flow/cfgspec/globalcfg.py", line 1909, in load
    self._load(fname, conf_type)
  File "/home/USER/miniconda3/envs/cylc-8.2.0/lib/python3.11/site-packages/cylc/flow/cfgspec/globalcfg.py", line 1889, in _load
    self.loadcfg(fname, conf_type)
  File "/home/USER/miniconda3/envs/cylc-8.2.0/lib/python3.11/site-packages/cylc/flow/parsec/config.py", line 80, in loadcfg
    sparse = parse(
             ^^^^^^
  File "/home/USER/miniconda3/envs/cylc-8.2.0/lib/python3.11/site-packages/cylc/flow/parsec/fileparse.py", line 565, in parse
    flines = read_and_proc(fpath, template_vars, opts=opts)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/USER/miniconda3/envs/cylc-8.2.0/lib/python3.11/site-packages/cylc/flow/parsec/fileparse.py", line 404, in read_and_proc
    odir = os.getcwd()
           ^^^^^^^^^^^
FileNotFoundError: [Errno 2] No such file or directory

Expected Behaviour

Current directory not existing should not cause failure in many commands.

Potential Fix

For the cylc scan case at least, the below snippet removes the failure and produces correct results. I cannot comment if this change would have flow on impacts to other commands.

$ diff fileparse.py fileparse.modded.py
404c404,407
<     odir = os.getcwd()
---
>     try:
>         odir = os.getcwd()
>     except FileNotFoundError:
>         odir = None
503c506,507
<     os.chdir(odir)
---
>     if odir is not None:
>         os.chdir(odir)

Although there are two error messages still printed to command line which I'm unsure the origin of and ideally wouldn't exist either.

shell-init: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory                                                         
shell-init: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory

[hjoliver]

Huh, good spotting.

That getcwd() call was introduced very recently: we now move to the workflow directory during file parsing, to make workflow files easily available to the template processor. Then we change back to the original after, just in case it matters.

[ColemanTom]

Huh, good spotting.

I'm very often at the moment doing cylc clean for a workflow I'm currently in the directory of whilst I'm experimenting with some things - preferring to use completely clean installs for this experimentation.

[oliver-sanders]

Note that changing the directory back to the original location afterwards is good practice, but not strictly necessary as it only applies to the Python shell that the Cylc command runs in, not the shell that launched the command, so we are safe to ignore this error.

[hjoliver]

We could put @ColemanTom 's fix in, but I've discovered it is not sufficient at least in some environments: in a Python venv on my laptop VM under WSL, and on the NIWA HPC, cylc commands fail long before then in pkg_resources init if my pwd does not exist.

[ColemanTom]

I only tested it in one environment (conda based) with one command (cylc scan). I'm sure it doesn't cover everything, I don't have that much free time to explore non-critical edge cases.

[hjoliver]

Well, let's merge your fix at least. (I've posted it as a PR).

[oliver-sanders]

cylc commands fail long before then in pkg_resources

All Cylc commands are loaded via pkg-resources so I would expect this to fail for all commands if it fails for any.

[hjoliver]

It fails for all commands, for me. But Tom's post suggests it (with his code tweak) doesn't fail in his environment?