E3SM-Project
[WIP] adds support st archive
for now to start a discussion only... need some actual testing with a small case
- @AaronDonahue: the st archive PR is here (not finished yet)
- @PeterCaldwell: this is potentially leadership material in terms of naming, etc. (I simply formatted things that will work in the current status quo, didn't choose names (scream) for any other purpose)
- @rljacob: for awareness (in xref https://github.com/E3SM-Project/E3SM/issues/6190, you said the eamxx team should add "eamxx" in the cases, but I thought that would involve more conversations and logisitcs, so I opted for a simpler solution that will work sooner
- @chengzhuzhang: you opened https://github.com/E3SM-Project/E3SM/issues/6190 so just letting you know
We probably need a signoff from Jim and Luca, so requesting reviews from them.
I removed myself as a reviewer, as I don't know much about this stuff. I think Jim/Rob can give better feedback.
An update:
I've been exploring manual alternatives to this, and I think they work fine.
It's difficult to finalize this PR without introducing some arbitrary semantics into the EAMxx IO system (at the very least, we likely need to inject something like ".h." in the pure output to differentiate it from restarts-oriented ones like ".rhist." and ".r.").
Another caveat --- st_archive will likely be extremely slow with EAMxx; it is already not really fast with ne30pg2 restart files; so imagine how slow it will be with ne1024pg2 files. I can tell you purely copying one of those restart files (2.2 TB) is borderline a nightmare.
I like what Peter suggested once: why don't we do what we like about st_archive by default in EAMxx? I think this is a great idea. I will workshop it with Luca (for internal IO) as well as Andrew (for rpointer stuff). I think we can make it work... so that's another pathway forward.
@mahf708 thank you for taking the initiative on this effort!
Originally, I thought it is straightforward to extend st_archive through CIME to cover eamxx files. Based on your exploration, it requires changing eamxx output file names. In addition, there is potential performance issue. The slowness is mostly from copying large size files, right? I think folks have been using move (--force-move ), which is much faster.
Closing, because now this has a conflict, and we are going to reconcile the repos completely soon. The conflict is due to https://github.com/E3SM-Project/scream/pull/2957.
Additionally, I think we will be doing something conventions-related in the IO task collection led by Aaron. So, we will deal with this as a subtask within the larger IO epic! Stay tuned!