Grid info tracking

[dschwoerer]

We do currently do some tracking of grid infos [1]

I would like to do the same for zoidberg - should I just duplicate the code, or should we generalise the code? e.g. hypnotoad_version -> grid_software_version? @johnomotani any thoughts?

[johnomotani]

My vote would be just duplicate the code to add whatever things you want for zoidberg. I don't have strong feelings though - it feels a bit wrong not to separate the grid generator metadata stuff out into its own method or something, but in practice we're unlikely to have more than hypnotoad and zoidberg to look after so on balance just adding some lines for zoidberg seems like the simplest option to me.

Edit: I vote for 'duplicate' because I think it's nice to be sure that if I'm checking hypnotoad_version it's definitely hypnotoad that's referred to.

[ZedThree]

+1 for duplicate for now, we could always refactor later. Ideally, one might imagine we had something like a list of input_software metadata with software_name, software_version, etc., so that we could record all the useful info for all the software that generated inputs.

I would say we should probably wrap this pattern up into a helper function:

  if (this->get(grid_id, "grid_id") == 0 and not grid_id.empty()) {
    file.add(grid_id, "grid_id", false);
  }

[dschwoerer]

Thinking about it again - would it not be best to include the whole grid file? For FCI the grid file is anyway needed for plotting. However, I am not sure what the best approach would be for this ...

We could copy the grid file as ${grid_id}.nc to the output folder? Then it would be sufficient to copy the grid_id and when xemc3.open_boutdataset() could collect the grid.

~~Btw, the default values should probably be changed:~~ (please ignore, I was reading it wrong)

[johnomotani]

@dschwoerer I agree the grid file needs keeping. At the moment it is up to the user to keep track of this - grid_id was just intended to be able to check that you do have the right grid file. I'm not sure always copying the grid file to the output file is a good idea - for our workflow this would lead to hundreds of copies of a few grid file floating around. If it suits your workflow, I guess it shouldn't be too hard to make a copy of the grid file from your PhysicsModel?

In the long term, including either the grid file itself (as a NetCDF group?), or all the info from the grid file, in the dump files seems like a good idea. When I've thought about it it seems like a pain to implement at the moment: the current DataFile interface is not great for this (e.g. needing to pass persistent Field3D variables to be saved, although I guess it might be possible to work around this...); and xarray doesn't handle NetCDF groups nicely (yet!). So seems to me like something to bear in mind when refactoring the file I/O rather than something to worry about now.

[ZedThree]

The IO refactor in #2209 should be able to handle this, although I'm not sure it's a good idea, as it would increase the file size of every output file. On the other hand, each output file would presumably keep its local chunk of the grid, which would be nice?

So, maybe make it a user option?