Gauges: binary output and thread safe version?

[rjleveque]

Gauge output is currently printed to an ascii file in a way that requires a critical block, since all gauges are written to the same file.

We might consider an option in which an array is allocated for each gauge and values are inserted in this rather than being written each timestep. Then write at the end of the run.

Advantages:

• Would not need critical block
• Should be faster than writing every time step for long runs with many gauges, particularly in binary
• Could also write in HDF or NetCDF form, perhaps useful.

Disadvantages:

• Don't know at start how many timesteps will be printed, and this can vary between gauges depending on where then are relative to refinement and what time interval is specified for printing. So we would need some reallocation strategy or else a way to dump partial results.
• Cannot view partial gauge outputs while run is still going, often quite useful in GeoClaw.

Note: could also write every time step but to a different file for each gauge, so wouldn't need the critical block. This might be advantageous for other reasons in general, but it might lead to a huge number of files in some cases. (Usually there are a few gauges at most, but might want to put down e.g. a 100 by 100 grid of gauges for some application?)

See also #94 and consider adding precision option.

[mandli]

I like the idea of having a separate file for each gauge as this would simplify loading the gauge data as well although if someone did do a 100 x 100 grid of gauges this would clearly be a problem (but maybe we should encourage fixed grid output then anyway). I would suggest that we set a buffer length for each gauge then and when that buffer fills (or gets close) spawn a new thread that would write the data out to a file. This would get rid of the critical block and allow for the calculation to continue.

[mjberger]

would spawning a new thread work with the usual way we operate, using the set num threads environment variable of openmp, and then using all of them? If you spawn a new one, what is left for the rest - or does it just rejoin the pool when it is down with i/o?

Marsha

On Dec 28, 2014, at 5:14 PM, Kyle Mandli [email protected] wrote:

I like the idea of having a separate file for each gauge as this would simplify loading the gauge data as well although if someone did do a 100 x 100 grid of gauges this would clearly be a problem (but maybe we should encourage fixed grid output then anyway). I would suggest that we set a buffer length for each gauge then and when that buffer fills (or gets close) spawn a new thread that would write the data out to a file. This would get rid of the critical block and allow for the calculation to continue.

— Reply to this email directly or view it on GitHub.

[mandli]

Having more threads spawned than what the number set by OMP_NUM_THREADS limits only the maximum number of active threads allowed to my understanding. The operating system should switch in and out thread contexts when it deems necessary and will eventually let the thread complete. The main win here would be that three of the threads could continue while one is used for writing. The other option here is to transition to the newer task based paradigm within OpenMP which fits this much more directly.

[mjberger]

My question was only about the priority of the threads - wouldn’t we rather have a thread continue integrating first, if that’s what it’s up to, then finish writing during a serial part of the code? How to make that happen?

Marsha

On Dec 31, 2014, at 2:34 PM, Kyle Mandli [email protected] wrote:

Having more threads spawned than what the number set by OMP_NUM_THREADS limits only the maximum number of active threads allowed to my understanding. The operating system should switch in and out thread contexts when it deems necessary and will eventually let the thread complete. The main win here would be that three of the threads could continue while one is used for writing. The other option here is to transition to the newer task based paradigm within OpenMP which fits this much more directly.

— Reply to this email directly or view it on GitHub.

[mandli]

I am not aware of a way in OpenMP to have that kind of fine grained control over the threads. There is OMP_WAIT_POLICY=passive which causes inactive threads to sleep faster but I am not sure if that would really accomplish what we want. I think the general feeling in the OpenMP community is that it's better to allow the OS schedule the threads rather than mucking about with those mechanisms. Besides, wouldn't this risk the potential for never getting the writing done?