EGSnrc
EGSnrc copied to clipboard
I/O bottleneck for EGS_ScoringSingle and EGS_ScoringArray
Reopening this as an issue: originally posted as issue #605 and then transferred to Discussion #688.
There are 2 pull requests related to this issue (#606, #707) and a suggested improvement on branch fix-egspp-scoring-speed
(extracted from #707). Note also a somewhat related pull request for sparse matrix storage in DOSXYZnrc (#405).
I ran into this issue myself a couple weeks ago with a 250x250x250 voxels XYZ geometry: simulation for a few histories took 0.03 seconds, but data I/O took 30 seconds! This makes simulation with large phantoms (in terms of number of voxels) completely impractical, just on account of of writing data one voxel at a time in text mode.
Discussed in https://github.com/nrc-cnrc/EGSnrc/discussions/688
Originally posted by MartinMartinov June 1, 2020 I've been running simulations computing 3ddoses in 512x512x117 phantoms, where the actual particle simulation runs pretty quickly, but the all the storeState/setState calls (I suspect) are bottlenecking the simulation significantly. I'm currently setting nbatch to 1 to avoid too many state calls which works well interactively, but when running several simulations in parallel accessing the same hard drive, it is still a huge efficiency killer when they all go to write egsdat files at once (once again, just a suspicion).
Would it be possible to write a parallel set of store/setState which save data in binary format (ie, storeBinState and setBinState) which usercodes that have massive arrays can use instead? I don't mind giving it a try myself, I'm just not sure how well it would fit into the current architecture, so I wanted to post here first.