[FEAT] hdf5 Nexus/CXI DB compatible Input/Ouput

[sjleake]

Is your feature request related to a problem? Please describe. handle the static metadata in a sensible way

Describe the solution you'd like An object to store all of the parameters of interest, this would require a converter for each beamline and would reduce the number of defined parameters required. I like the idea of a config file overwrite where if defined it is overwritten.

[sjleake]

for the diffractometer definition

https://manual.nexusformat.org/classes/base_classes/NXtransformations.html#nxtransformations

[DSimonne]

I will use this for the GUI file input/output, looking at the nexus format, as soon as I have time =)

[DSimonne]

Hi, I could use some help with this, it's really painful to code There is an example file in /data/id01/inhouse/david/TestFileFormat I still need to add some units but if you could check the file and tell me is it corresponds to the idea behind the cxi files ?

[sjleake]

Hi, It is looking really good! Just reading the file it is clear what has been done, which is exactly the reason to work this way. I didn't see diffractometer motors yet or the transformation matrix option (this was requested by Vincent for PyNX). Not sure if the raw data and the phased data need to be separated. From experience it is easy to corrupt hdf5 files so it is often advantageous to keep them isolated.

[DSimonne]

The idea was to keep a single file for all the data, I basically add everything to a cxi file generated by PyNX. Idk if we should separate it or not :/ Because if we agree on one file, it is easy to share, and also easy to load back in the GUI :/ For the transformation matrix, he meant the one used for orthogonalisation ? That can easily be done. The diffractometer motors meaning the motor used for the rocking curve ?

[sjleake]

I am tempted to say raw diffraction data + metadata should never be touched. Risk of corruption is not worth it. This is the file generated by the beamline. If a copy is to be placed in the file with the analysis (I guess this is OK) it should be a single copy, that each entry points to. Otherwise the difference between 1 and 2 files is not large, you can make the analysis file (master) point at the data file so it is effectively transparent to the GUI. Yes regarding transformation matrix. Yes all relevant motors, filters, counters etc. In the future our data will provide a full description of the beamline at the time of data collection.

[DSimonne]

Actually yes I copy the raw diffraction data file and create a new one so in the end there are a few files:

• the raw diffraction file, for example at SixS a NeXuS file, at ID01 the raw data is left untouched in its directory
• the pre-processed diffraction file saved as .cxi by PyNX, this one depends on the pre-processing parameters such as the binning or cropping
• the final .cxi file, it is basically the same cxi file with more data entries that detail the analysis, this is the one I sent you

So we separate the raw data, the cxi file representing the pre processed data, and the final file with the analysis detail enclosed.

Regarding the transformation matrix, this can be saved in the final cxi file after post-processing

For the motors, these are already stored in a NeXuS file at SixS. I suppose that what you mean is that in the future ID01 will also have the same kind of NeXuS file with all the beamline detail. Currently the final file only encloses detail about the parameters relevant for the analysis, such as the central pixel position, the inplane or outofplane rocking curve, etc ... But nothing about parameters that are not used in the analysis such as the monochromator, the nb of filters etc ... This would be difficult to support with so many beamlines ...

[DSimonne]

I compiled a script that allows to compute the transformation matrix from the parameters used in strain.py. It's available in the GUI and automatic. The transformation matrix is saved in the final cxi file.