SCOREC
Proper procedures to load partitioned mesh
Currently, the following procedures are used to load partitioned mesh in both XGCm and GITRm following PUMIPic example:
- read in the full mesh
- load in the partition file and create PICparts
- delete the full mesh as necessary
These procedures are not optimal at all and will encounter memory issue as the mesh size becomes large (>20-50 million mesh elements for example, depending on the GPU memory). They are similarly to this test:
https://github.com/SCOREC/pumi-pic/blob/b6678b0a0b8c9ad1e143831bdc2920b944b0f5ff/test/test_full_mesh.cpp#L22-L34
The proper way would be to read in the partitioned mesh directly. It's not clear if below function is already providing the necessary steps: https://github.com/SCOREC/pumi-pic/blob/b6678b0a0b8c9ad1e143831bdc2920b944b0f5ff/src/pumipic_file.cpp#L118-L206 The above function is used in the following test: https://github.com/SCOREC/pumi-pic/blob/b6678b0a0b8c9ad1e143831bdc2920b944b0f5ff/test/pseudoXGCm.cpp#L379-L388
I thought I had remembered using pre-constructed meshes for performance testing. The read(...) function you found looks correct and the write(...) function to produce the constructed mesh is also in src/pumipic_file.cpp.
This doesn't solve the high memory usage issue, but may give you a chance to run the mesh construction job on nodes (possibly using a CPU only build) with more memory as a pre-processing step before running an analysis.
@cwsmith: yes, looking further we need to write the partition file *.ppm file in a node with more memory to get the partitioned mesh and partition information.
I believe @cwsmith is right that we pre-constructed the parallel meshes for performance testing. It was most likely done on the high-memory CPU nodes in the CCI at the time. The read and write methods should be reusable for your use-case.
