cime in maint-2.0 failed to work on anvil/chrysalis

[wlin7]

Creation of a v3.LR.amip case (F20TR) on either chrysalis or anvil first got the following errors.

Running /gpfs/fs1/home/ac.wlin/ACME/integration/E3SM.maint-2.0/components/eam//cime_config/buildnml
Traceback (most recent call last):
  File "/gpfs/fs1/home/ac.wlin/ACME/integration/E3SM.maint-2.0/components/eam//cime_config/buildnml", line 239, in <module>
    _main_func()
  File "/gpfs/fs1/home/ac.wlin/ACME/integration/E3SM.maint-2.0/components/eam//cime_config/buildnml", line 234, in _main_func
    caseroot = parse_input(sys.argv)
               ^^^^^^^^^^^^^^^^^^^^^
  File "/gpfs/fs1/home/ac.wlin/ACME/integration/E3SM.maint-2.0/cime/CIME/buildnml.py", line 28, in parse_input
    args = parse_args_and_handle_standard_logging_options(argv, parser)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/gpfs/fs1/home/ac.wlin/ACME/integration/E3SM.maint-2.0/cime/CIME/utils.py", line 1687, in parse_args_and_handle_standard_logging_options
    _check_for_invalid_args(args[1:])
  File "/gpfs/fs1/home/ac.wlin/ACME/integration/E3SM.maint-2.0/cime/CIME/utils.py", line 2478, in _check_for_invalid_args
    if get_model() != "e3sm":
       ^^^^^^^^^^^
  File "/gpfs/fs1/home/ac.wlin/ACME/integration/E3SM.maint-2.0/cime/CIME/utils.py", line 530, in get_model
    cime_config = get_cime_config()
                  ^^^^^^^^^^^^^^^^^
  File "/gpfs/fs1/home/ac.wlin/ACME/integration/E3SM.maint-2.0/cime/CIME/utils.py", line 304, in get_cime_config
    _CIMECONFIG = _read_cime_config_file()
                  ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/gpfs/fs1/home/ac.wlin/ACME/integration/E3SM.maint-2.0/cime/CIME/utils.py", line 265, in _read_cime_config_file
    cime_config = configparser.SafeConfigParser()
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AttributeError: module 'configparser' has no attribute 'SafeConfigParser'. Did you mean: 'RawConfigParser'?

When replacing cime_config = configparser.SafeConfigParser(), with cime_config = configparser.ConfigParser() it could go a little further but would then fail with

Traceback (most recent call last):
  File "/lcrc/group/e3sm/ac.wlin/E3SM_simulations/E3SMv2/v2.LR.amip_0101_br2015/case_scripts/./case.build", line 256, in <module>
    _main_func(__doc__)
  File "/lcrc/group/e3sm/ac.wlin/E3SM_simulations/E3SMv2/v2.LR.amip_0101_br2015/case_scripts/./case.build", line 240, in _main_func
    success = build.case_build(
              ^^^^^^^^^^^^^^^^^
  File "/gpfs/fs1/home/ac.wlin/ACME/integration/E3SM.maint-2.0/cime/CIME/build.py", line 1286, in case_build
    return run_and_log_case_status(functor, cb, caseroot=caseroot)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/gpfs/fs1/home/ac.wlin/ACME/integration/E3SM.maint-2.0/cime/CIME/utils.py", line 2447, in run_and_log_case_status
    rv = func()
         ^^^^^^
  File "/gpfs/fs1/home/ac.wlin/ACME/integration/E3SM.maint-2.0/cime/CIME/build.py", line 1270, in <lambda>
    functor = lambda: _case_build_impl(
                      ^^^^^^^^^^^^^^^^^
  File "/gpfs/fs1/home/ac.wlin/ACME/integration/E3SM.maint-2.0/cime/CIME/build.py", line 1180, in _case_build_impl
    _build_model_cmake(
  File "/gpfs/fs1/home/ac.wlin/ACME/integration/E3SM.maint-2.0/cime/CIME/build.py", line 472, in _build_model_cmake
    cmp_cmake_args += _create_build_metadata_for_component(
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/gpfs/fs1/home/ac.wlin/ACME/integration/E3SM.maint-2.0/cime/CIME/build.py", line 927, in _create_build_metadata_for_component
    buildlib = import_from_file(
               ^^^^^^^^^^^^^^^^^
  File "/gpfs/fs1/home/ac.wlin/ACME/integration/E3SM.maint-2.0/cime/CIME/utils.py", line 25, in import_from_file
    loader = importlib.machinery.SourceFileLoader(name, file_path)
             ^^^^^^^^^^^^^^^^^^^
AttributeError: module 'importlib' has no attribute 'machinery'

Does this mean we need to update the cime submodule for maint-2.0? prod_maint-2.0 tests on anvil has been fine, though.

[rljacob]

Are you sure your clone has the right submodules for maint-2.0?

[wlin7]

I would think so. A case created off it in the summer ran ok. Yesterday when trying to run some xml commands from its case_scripts directory, it reported failure. Creation of new case from the same code base reported the same error. The problem does not go away after forcing to deinit cime then reinitialize.

I am trying a fresh clone of maint-2.0 now.

[wlin7]

No luck with a fresh clone of maint-2.0.

My test script is: /home/ac.wlin/E3SM/Cases/testing/scripts/run.v2.LR.amip_0101_br2015.sh

[rljacob]

Try running one of the production tests that are passing in the nightlies.

[mahf708]

I tried reproducing this issue to no avail. The likeliest explanation is that some submods were not updated correctly?

@wlin7 are you still seeing this issue? I can try to help

[wlin7]

Hi @mahf708 , thanks for offering to help. Yes, the problem still exists. The branch can still work on pm-cpu.

On chrysalis, I did a test swapping in the latest cime. It could go much further, but then failed in building a share module (related to sea ice)

[evasinha-pnnl]

@wlin7 I was running into the same error on compy today when submitting a case using E3SM v2.1 branch. I tried updating the submodules several times, but the issue stayed unresolved. Finally, I realized that I had Python 3.12.3 loaded. I unloaded it and loaded module load python/anaconda3-2020.02 (version Python 3.7.6). I was then able to create a case.