Need to test the integration in the of post-processing into CIME

[bertinia]

Test the env_batch.xml addition in an existing CASEROOT with an existing postprocess directory.

Add documentation for how to setup the env_batch.xml.

[bertinia]

There are some real concerns with state management in the linear dependency and job submission associated with the CIME workflow. Here's one possible problem scenario that Sheri posses:

I thought of another issue that users could encounter that the CIME workflow will have to handle. It is related to the issue we were working through after the workflow meeting. The resolution was to set a flag (XML?) that would signal that the date range for that diagnostic had already ran and not to try running it again. My thought is a scenario where the diag average script waits in the queue for a long time and then eventual runs. My question is how is CESM resubmitted? If the short-term archiver submits CESM than potentially you'll have two (or more if the queue is really backed up) diag average jobs in the queue. What if the wait is so long that one of the diag avg script is still running when the next CESM finishes, than the archiver finishes, and then the second diag avg script starts to run because the finished flag hasn't been set yet. Do we also need a 'lock' flag? My thinking is it would have to be checked and then set in the same spot right after it checks to see if the data is there and can continue. This is still a potential race condition, but is smaller. Or is it that the last job always resubmits CESM and this won't be a problem? My thought is that the BGC portion of the diags takes a long time to run (even with parallelization) and this could be a problem even if the job is submitted to geyser without that long of a queue wait."

It's not clear how much extra postprocessing state info and logic would need to be maintained in additional XML variables, state files, and script logic. Is it worth it considered we have an end-to-end workflow manager with cylc and our users can currently manually submit postprocessing scripts as required for their workflow and/or use the postprocessing tools (pp_config, copy_html) in their own scripts to create a customized workflow?

I'm going to place this issue at a lower priority for now unless early adopters would like to have it available sooner than later.