Baseline archiving fails if regression tests are re-run manually

[nusbaume]

What happened?

If, after running test_driver.sh, one finds that a test needs to be re-run manually and does so, then when they try to run archive_baseline.sh it can fail to actually archive any baselines, although one may not notice that it failed as it still creates all of the relevant directories and doesn't throw any sort of warning or error message.

What are the steps to reproduce the bug?

Follow the steps listed here for running the CAM regressions tests:

https://github.com/ESCOMP/CAM/wiki/CAM-SE-Workflows#run-cam-tests

Then try doing the steps for manually re-running individual tests (e.g. ./.case_test --reset), then try archiving the baselines. If the bug triggered then you should find that the test which you re-ran manually hasn't actually been archived, even if the test itself passed.

What CAM tag were you using?

cam6_3_137

What machine were you running CAM on?

CISL machine (e.g. cheyenne)

What compiler were you using?

Intel

Path to a case directory, if applicable

No response

Will you be addressing this bug yourself?

Any CAM SE can do this

Extra info

For now, if you find that you manually re-ran a test and the baseline isn't archiving properly, then you can either try and manually archive the baseline for that test yourself, or re-run the test suite from the beginning, after which the baseline archiving script should work as expected.

Finally, pinging @fischer-ncar in case he has any insights.

[fischer-ncar]

@nusbaume how repeatable is this bug? This doesn't sound like an issue with archive_baseline.sh, but with the cime scripts. Are the baselines being created in $CESM_TESTDIR/baselines?

[nusbaume]

@fischer-ncar Good question! Let me see if I can make a simple test workflow to see if the problem is fully reproducible. I'll let you know if I get something working.