Output filename extensions

[ketch]

Note: this is now a CEP and should be discussed at https://github.com/clawpack/clawpack/issues/51.

We should probably use a different filename extension for each of the output file types supported. Right now, any file output using PETSc is name *.ptc (except for VTK files). Also, something we inherited from Clawpack that doesn't really make sense to me is that the file number is in the extension.

I propose the following:

• metadata pickle: claw####.pickle (for both PyClaw and PetClaw output)
• PETSc binary: claw####.ptc
• VTK: claw####.vtk (same as current)
• HDF5: claw####.h5 (for both PyClaw and PetClaw output)
• NetCDF: claw####.nc (for both PyClaw and PetClaw output)
• ASCII: claw####.txt (for both PyClaw and PetClaw output)

All aux files would have the same names as the q files, but with the claw prefix replaced by aux.

Since HDF5 files (for instance) that are written with PETSc are probably not readable with the PyClaw HDF5 I/O routine, we could add the file format specification (including whether it came from petclaw or pyclaw) to the metadata pickle. Alternatively, we could add a "pet" prefix to the extension for petclaw output, for instance. Either way, that would make it so that the user would only need to specify an output directory when loading data; the output format would be detected automatically (from the filename or the metadata).

Note that it would no longer make sense to write data from two different runs to a single output directory, unless the user manually modified the filename prefix. That's because the metadata pickles always have the same names. I think this is a good thing.

This proposal would completely get rid of the fort prefix, which I think also doesn't have any useful meaning anymore.

[mandli]

I strongly support a move towards a scheme like this and would hazard a push even to perhaps consider it a broader move for all of the Clawpack packages. Couple of small comments:

• We should avoid backwards compatibility issues by maintaining the old file defaults until we do a major version bump.
• For ASCII output I might suggest .data or .dat although I am not sure if it really matters to me. Might be better to keep .txt because most sane machines on the planet know what to do with those.
• I did some searching for the appropriate suffix for the pickle files and found a lot of .pickle suggestions. Maybe we should go with the longer, more descriptive version?
• As a follow up to my suggestion to broaden the application to all of Clawpack, do we want to hash out a proposal here and move it to claw-dev? Alternatively we can change PyClaw and it's ilk for right now and point out how awesome it is.

[ketch]

We should avoid backwards compatibility issues by maintaining the old file defaults until we do a major version bump.

Agreed. But VTK files already use the proposed format, and we didn't have working HDF5 or NetCDF until now, so I propose immediately going with the new names for those if that doesn't seem too confusing.

For ASCII output I might suggest .data or .dat although I am not sure if it really matters to me. Might be better to keep .txt because most sane machines on the planet know what to do with those.

I favor .txt rather strongly for the reason you mention. Users would be able to double click the file and have it open in something reasonable (or quick-look it on Mac OS, etc.) I'm willing to hear reasons for using something else.

I did some searching for the appropriate suffix for the pickle files and found a lot of .pickle suggestions. Maybe we should go with the longer, more descriptive version?

Okay, I updated the proposal above.

As a follow up to my suggestion to broaden the application to all of Clawpack, do we want to hash out a proposal here and move it to claw-dev? Alternatively we can change PyClaw and it's ilk for right now and point out how awesome it is.

Great, I wanted to propose that but was waiting to see what others said. Let's come up with something we agree on and propose it on claw-dev. If there is resistance, we can still implement it for PyClaw. How do you want to name the various formats that can be output by the other packages? And do we want the filename to distinguish which package it came from?

Also, .nc is apparently standard for NetCDF, but for hdf5 I see both .h5 and .hdf5. Any preference?

[mandli]

We should avoid backwards compatibility issues by maintaining the old file defaults until we do a major version bump.

Agreed. But VTK files already use the proposed format, and we didn't have working HDF5 or NetCDF until now, so I propose immediately going with the new names for those if that doesn't seem too confusing.

There is work being done in GeoClaw to support HDF5 and NetCDF although I think we should not wait for that to get finished up.

For ASCII output I might suggest .data or .dat although I am not sure if it really matters to me. Might be better to keep .txt because most sane machines on the planet know what to do with those.

I favor .txt rather strongly for the reason you mention. Users would be able to double click the file and have it open in something reasonable (or quick-look it on Mac OS, etc.) I'm willing to hear reasons for using something else.

Sounds good.

I did some searching for the appropriate suffix for the pickle files and found a lot of .pickle suggestions. Maybe we should go with the longer, more descriptive version?

Okay, I updated the proposal above.

As a follow up to my suggestion to broaden the application to all of Clawpack, do we want to hash out a proposal here and move it to claw-dev? Alternatively we can change PyClaw and it's ilk for right now and point out how awesome it is.

Great, I wanted to propose that but was waiting to see what others said. Let's come up with something we agree on and propose it on claw-dev. If there is resistance, we can still implement it for PyClaw. How do you want to name the various formats that can be output by the other packages? And do we want the filename to distinguish which package it came from?

If the formats are the same I am not sure if we need to really. Originally the prefix could be tailored this way when a solution writes itself out but I never really saw a reason behind doing it really.

Also, .nc is apparently standard for NetCDF, but for hdf5 I see both .h5 and .hdf5. Any preference?

I tend to prefer the more descriptive suffixes, hopefully in this day and age operating systems will not choke on a file suffix length > 3.

[ketch]

I tend to prefer the more descriptive suffixes, hopefully in this day and age operating systems will not choke on a file suffix length > 3.

I was thinking the same, but @sanromd tells me that Paraview has a strong preference for the .h5 suffix. It seems that .h5 is the official extension (and .hdf actually means hdf4 or perhaps hdf<=4). See http://www.hdfgroup.org/HDF5/doc1.6/Tools.html.

[mandli]

Looks like that is the case. I fine with that then.