Modified config_archive.xml to archive CAM+DART files.

[kdraeder]

This would be a fix for issue #1301 I'll provide background or details as needed.

Closes #1301

[cecilehannay]

I did a few runs to test the new code and I see an issue with the code when more frequent initial conditions are required (for instance I use: inithist = 'MONTHLY')

First I did a run with the default inithist, and everything seems fine:

• In: /glade/derecho/scratch/hannay/archive/f.e30_cam6_4_089.FHISTC_LTso.ne30.baseline156.001 the *.i.* are both rest in atm/hist.
• while in: /glade/derecho/scratch/hannay/archive/f.e30_cam6_4_089_raeder.FHISTC_LTso.ne30.baseline156.001 the *.i.* are only rest. I am fine with this

But if I use inithist = 'MONTHLY'

• In: /glade/derecho/scratch/hannay/archive/f.e30_cam6_4_089.FHISTC_LTso.ne30.baseline156.002 the extra *.i.* are in atm/hist.
• while in: /glade/derecho/scratch/hannay/archive/f.e30_cam6_4_089_raeder.FHISTC_LTso.ne30.baseline156.002 the extra *.i.* are not moved to the archive directory in they stay in the run directory. This is an issue.

[cacraigucar]

I did a few runs to test the new code and I see an issue with the code when more frequent initial conditions are required (for instance I use: inithist = 'MONTHLY')

First I did a run with the default inithist, and everything seems fine:

• In: /glade/derecho/scratch/hannay/archive/f.e30_cam6_4_089.FHISTC_LTso.ne30.baseline156.001 the *.i.* are both rest in atm/hist.
• while in: /glade/derecho/scratch/hannay/archive/f.e30_cam6_4_089_raeder.FHISTC_LTso.ne30.baseline156.001 the *.i.* are only rest. I am fine with this

But if I use inithist = 'MONTHLY'

• In: /glade/derecho/scratch/hannay/archive/f.e30_cam6_4_089.FHISTC_LTso.ne30.baseline156.002 the extra *.i.* are in atm/hist.
• while in: /glade/derecho/scratch/hannay/archive/f.e30_cam6_4_089_raeder.FHISTC_LTso.ne30.baseline156.002 the extra *.i.* are not moved to the archive directory in they stay in the run directory. This is an issue.

@kdraeder - Are you going to supply a fix for this? If not, do you no longer need this PR to go into CAM?

[kdraeder]

@cacraigucar I was just working on this!

@cecilehannay Thanks for running those tests.

In the first pair (inithist = default)

• the last .i. file is still in $rundir and in rest/1996-01-01-00000, which is good for DART.
• The default code also copies it to atm/hist, while the raeder code does not.
• In the default code the intermediate file (1995-01-01) is not in rundir, but is in the rest/1995-01-01-00000 directory. It is also in atm/hist. So it's treated as a restart and a history file.

In the last test (inithist = MONTHLY, raeder's archive mods, run for 4 years instead of 2)

• the .i. files from the restart time (the last date of the run) are copied to the rest archive, and are left in $rundir. This is the functionality that DART needs.
• As Cecile points out, the intermediate .i. files are not copied or moved.

How are the intermediate .i. files used?
If it's only to start runs, then it seems that atm/hist is not the right place for them.
If they're also treated like history files to be statistically analyzed like .h#. files, then the best place to store them is ambiguous, and there's a conflict with DART's use of them. When inithist is not MONTHLY, then they are archived in 2 places, which satisfies all the hoped-for uses. So the question seems to be "Can we make MONTHLY (and shorter) behave the same way?" If someone can point me to the code where this is handled, it would speed up my attempt to find the way.

[cacraigucar]

@cacraigucar I was just working on this!

@cecilehannay Thanks for running those tests.

In the first pair (inithist = default)

• the last .i. file is still in $rundir and in rest/1996-01-01-00000, which is good for DART.
• The default code also copies it to atm/hist, while the raeder code does not.
• In the default code the intermediate file (1995-01-01) is not in rundir, but is in the rest/1995-01-01-00000 directory. It is also in atm/hist. So it's treated as a restart and a history file.

In the last test (inithist = MONTHLY, raeder's archive mods, run for 4 years instead of 2)

• the .i. files from the restart time (the last date of the run) are copied to the rest archive, and are left in $rundir. This is the functionality that DART needs.
• As Cecile points out, the intermediate .i. files are not copied or moved.

How are the intermediate .i. files used? If it's only to start runs, then it seems that atm/hist is not the right place for them. If they're also treated like history files to be statistically analyzed like .h#. files, then the best place to store them is ambiguous, and there's a conflict with DART's use of them. When inithist is not MONTHLY, then they are archived in 2 places, which satisfies all the hoped-for uses. So the question seems to be "Can we make MONTHLY (and shorter) behave the same way?" If someone can point me to the code where this is handled, it would speed up my attempt to find the way.

@brian-eaton and/or @jedwards4b - Can one of you direct @kdraeder to the code location where he might need to make his mods?

[jedwards4b]

Have you tried adding .i files to both restart and history? That is add back: <hist_file_extension>i..*.nc$</hist_file_extension>

[kdraeder]

My test using @jedwards4b suggestion results in copies of the intermediate .i. files in archive/atm/hist and final files in $rundir, archive/atm/hist, and archive/rest. So there's redundancy in the final files, but they aren't frequent or huge, so this looks like a reasonable and easy solution.

I actually tested <hist_file_extension>i\..*\.nc(\.gz)?$</hist_file_extension> to include the need for DART to handle compressed files and to make the pattern more restricted to '.i.' files. If that's not misguided, I'll include this form in a PR.

[jedwards4b]

@kdraeder sounds good, thank you.

[kdraeder]

If that's not misguided, I'll include this form in a PR.

I should have written "I'll commit this change and push it for review"

[kdraeder]

@cecilehannay @jedwards4b
I added a line to make st_archive save the .i. files as both history and restart. Github tells me 'These conflicts are too complex to resolve in the web editor.' but 'Changes can be cleanly merged.'

I don't see a way to see the conflicts. I believe the changes are just:

moving the h\d*.* line down (existing line 7 to new line9) I believe this is necessary because rest_file_extensions need to be before hist_file_extensions. adding a rest_file_extension line for i.\d.*.nc files. keeping the hist_file_extension for those same files.

Any ideas for resolving this?

[jedwards4b]

It wants you to do the merge manually in your sandbox and push it back up. The link above, View command line instructions, tells you exactly what you need to do.

[kdraeder]

"Command line" is not appearing in my view of this page, and I haven't found a way to make it appear. I can figure out how to merge, but it would be nice to be working in the same github world as the rest of CESM. Do you have any off-the-cuff suggestions?

[jedwards4b]

You don't see this?

[kdraeder]

No, I see this. I guess I don't have permission to merge, so it's not showing me the button and alternative.

[jedwards4b]

Okay - I think that I fixed it - your change to ChangeLog was premature. Just post the ChangeLog entry to a comment here and let the cam team add it.

[kdraeder]

It felt premature, but the contribution workflow instructions seemed clear about it. Thanks for fixing it, and sorry for the extra work!

I see that the config_archive.xml file has changes that I believe that I didn't make, so I'd like to confirm that they are intended. The rpointer file names in the old version used 'atm', while the new ones use 'cam'.
Will this be true for the other components?

Proposed Changelog entry:

---

One-line Summary: Make st_archive archive initial files as history and restart (for DART)

cime_config/config_archive.xml; Add a line to archive initial files as restarts.
Move the history file line to the history file group (after the restart file group). ? Change rpointer file names to use 'cam' instead of 'atm'

[jedwards4b]

This is correct, and only needed for cam. Just merged to the latest values.