mesa
mesa copied to clipboard
Reduce IO operations by using flush instead of repeated open/close.
This was suggested by Nick Carriero of the Flatiron Scientific Computing Core. Nick wrote to me with:
One quick thing to do would be to use flush rather than close, if the goal is to ensure data is written to disk. The metadata ops involved in closing and then reopening a file are much "heavier" than just reading from or writing to a file.
This came up because I was running many jobs on a cluster and the number of ops became an issue. Nick looked into it, noticed that 3/7 of those ops were involved in opening and re-closing history.data
, profiles.index
, and pgstar.dat
, each of which gets opened then closed once per step.
His suggestion is to open these files once at the start of a run, then use the Fortran FLUSH
intrinsic after our write statements are all done to ensure that all data is written to disk. This can happen once per step, and we just close the file handle at the very end.
I don't think this is urgent, and I have a workaround for my use case (just locating all of the files on a local scratch system then copying them back when the run finishes), but I didn't want to forget it because it seems like a good idea for a rainy day.