ESCOMP
ncar_pylib
looks like CISL is changing how they handle ncar_pylib. Not sure how this needs to change our workflow for python tools, but I wanted to raise the issue here to consider. Below is the error I received when activating ncar_pylib
**************************
*
* DEPRECATION NOTICE!
*
* The "ncar_pylib"-based NCAR Package Library has been deprecated in favor of
* the newer conda-based NPL. Please migrate your workflows to the conda solution
* as soon as possible. This older method and associated virtual environments
* will be removed from the system in the coming months. A specific date will be
* announced once it is known via the Daily Bulletin.
*
* See https://arc.ucar.edu/knowledge_base/83853599 for documentation on using the
* conda-based NCAR Python Library environments.
*
**************************
Yes, this is where using the conda python environment that we setup becomes important. We do have a conda text file for ctsm_py to run the tools under the python directory. I did add a bash script that helps at setting up that environment as well "manage_python_env" at the top level. There's more we need to do, and for example we want to change the name. But, we do have an initial setup that addresses this. The advantage of this is also that the conda environment is general and can be used on any system, including for example izumi.
Doing a "git grep ncar_pylib" I notice that most of this is just documentation. So that can be readily changed. I've started changing some of that for ctsm5.1.dev100, and we should do that more as we
One thing I did notice was the that the fsurdatmodifyctsm.py system test currently relies on ncar_pylib. I think what we should do is to do the following in the subprocess that calls the Python script:
"<make sure conda is loaded>; manage_python_env; conda activate ctsm_pylib; <run script>"
We talked about this some in our standup meeting this morning. Another solution is to have a wrapper around the test system that sets up the conda environment. Although that is problematic for the parts that are sent to batch nodes, as I don't know if the conda environment is inherited when sent to the batch machine.
This is mostly handled with the addition of the script to help build a conda environment. py_env_create. The thing left to do is remove the customization that fsurdat_modifier has for ncar_pylib.
@slevisconsulting I'd like to assign this to you to remove the customization that fsurdat_modifier has for ncar_pylib. Is that something you could do and appropriately charge to your current contracts? I assume you might need to make sure this is an appropriate use of your time.
@ekluzek fsurdat_modifier does not need ncar_pylib to work, so we should be all set. Current instructions in README.fsurdat_modifier state:
To run on Cheyenne/Casper/Izumi
1) (Un)load, execute, and activate the following:
module unload python
module load conda
./py_env_create
conda activate ctsm_py
(Use "deactivate" to reverse the latter.)
As we discussed this morning, these instructions won't work on izumi and my workaround there is to conda activate ctsm_py
OK, the one thing to do then @slevisconsulting would be to remove the code in the system test for it Right now this test invokes ncar_pylib from a specific path on cheyenne...
cime_config/SystemTests/fsurdatmodifyctsm.py
It should be changed to run py_env_create as you outline above.