Check to make sure env is set up properly

[sgoodm]

Primarily an issue with deployments, but could also cause issues across run modes.

Currently if the env is not set up properly (e.g., for the agent - loading conda env, setting tmpdir) there is a largely silent failure and Prefect will just hang. The only clue is that the logs from the failed dask-workers will have something along the lines of "could not find temp directory".

Ideally, we should recognize the env is not set up and throw a warning, and terminate the deployment before it gets started

[jacobwhall]

Here is some code to test if the current conda environment is correct, tested on a vortex compute node:

import os

logger = self.get_logger()

correct_env = "geodata38"

try:
    conda_env = os.environ["CONDA_DEFAULT_ENV"]
except KeyError:
    logger.warning("No conda environment detected! Have you loaded the anaconda module and activated an environment?")
else:
    if conda_env != correct_env:
        logger.warning(f"Your conda environment is {conda_env} instead of {correct_env}!")

[jacobwhall]

And here is some code to test if the current $TMPDIR environment variable points to within /local/, the default on HPC vortex nodes. I've tested this too

import os
from pathlib import Path

logger = self.get_logger()

try:
    tmp_dir = Path(os.environ["TMPDIR"]).resolve(strict=True)
except KeyError:
    logger.warning("No $TMPDIR environment variable found!")
except FileNotFoundError:
    logger.warning("$TMPDIR path not found!")
else:
    slash_local = Path("/local").resolve()
    for p in tmp_dir.parents:
        if p.resolve() == slash_local:
            logger.warning("$TMPDIR in /local, deployments won't be accessible to compute nodes.")

[jacobwhall]

@sgoodm are there other environment checks beyond #155 you'd like to add?

[sgoodm]

@jacobwhall This is looking good - we may want to put the conda env in the config so it is easily changeable

[jacobwhall]

we may want to put the conda env in the config so it is easily changeable

I agree we should make this configurable per-dataset. I'm hopeful my work on the "universal flow" (#146) could make thise easier to standardize across dataset scripts. If not, I can go back through them and add a conda_env variable to each run config.

Until then, I suggest we keep this issue open and merge #155