Scott Wittenburg

Hmmmm, I wonder: If rank X opens a file already opened and written by rankY, should it be doing that in append mode or else the file will get overwritten?

> If rank X opens a file already opened and written by rankY, should it be doing that in append mode or else the file will get overwritten? That was...

> But we're not finding the data that should be there. You say that this works with writing just one timestep. Is it possible that writing the 2nd timestep is...

> Are you closing files after the write responsibility shifts? If not I can imagine that if the responsibility shifts back you might need to close and re-open to get...

> > ``` > > 2. guarding against Bad Things when doing multiple Put() within a timestep > > ``` > > Just to clarify this one, multiple Put() shouldn't...

Well boo, I thought my zero blocks problem was solved, but it has cropped up again on a real HPC with a lustre filesystem. In that environment the problem can...

It occurred to me that one other difference between `EveryoneWritesSerial` and `DataSizeBased` aggregation _for a single timestep_ is that in the latter case, I'm currently calling `OpenFiles()` without a communicator...

Will we still need this [block](https://github.com/spack/spack/blob/develop/lib/spack/spack/ci/gitlab.py#L136-L155) after this PR is merged?

Might be worth checking with @eugeneswalker whether UO has the capacity to handle these extra builds.

I tried to reproduce a `develop` [failure](https://gitlab.spack.io/spack/spack/-/jobs/15087851) using this branch. Maybe I just took the perfectly wrong set of steps: first I tried just pasting the command from the trace...