dedalus icon indicating copy to clipboard operation
dedalus copied to clipboard

Fixes virtual file merge logic & removes deprecated distributed set merge logic

Open evanhanders opened this issue 2 years ago • 2 comments

This PR addresses #205 and updates post.py to reflect Dedalus' new output paradigm by removing deprecated functions for merging partial files. Functionality for merging partial files => complete files after a simulation is updated. Tests have been added for each of the new parallel output types ('virtual', 'mpio', 'gather').

All output and parallel output tests now pass. There are some failing ncc tests, but those should be unaffected by these commits.

evanhanders avatar Jun 20 '22 20:06 evanhanders

It should be possible to simply merge virtual datasets into concrete datasets within the existing set file using the merge_task method on the new H5VirtualFileHandler class. I think this should remove the need for most of the logic in post.py, including creating a new file, copying scales, etc.

kburns avatar Jun 22 '22 13:06 kburns

Hi Keaton, I didn't know about that method! I'll take a look and try to get that implemented. I've marked the PR as a draft for now.

evanhanders avatar Jun 22 '22 15:06 evanhanders