How to write proper tilt series EMD file?

[ercius]

I want to create an EMD file which will be recognized by tomviz as a tilt series and load the angles in the dim1 dataset. I see that in this line there is comparison for: dim1.attrs['units'] == '[deg]' Then it seems that tomviz will recognize the data as a tilt series.

I am unable to trigger this behavior using h5py and ncempy.io.emd. The EMD file will load but the file dimensions are backwards (Z, Y, X) and not (X, Y, Z), which is expected for a volume.

I know that special care needs to be taken with h5py and strings, but nothing I try (fixed length, variable length ASCII, etc.) works. I cant get the expected behavior. See these examples: fid['/data/tomography/dim1'].attrs['units'] = np.string_('[deg']) fid['/data/tomography/dim1'].attrs['units'] = b'[deg'] fid['/data/tomography/dim1'].attrs['units'] = bytes('[deg'],'ASCII') fid['/data/tomography/dim1'].attrs['units'] = (bytes('[deg'],'ASCII'), )

The last one seems to match the output of an EMD file written by tomviz using h5dump:

DATASET "dim1" {
            DATATYPE  H5T_IEEE_F32LE
            DATASPACE  SIMPLE { ( 74 ) / ( 74 ) }
            ATTRIBUTE "name" {
               DATATYPE  H5T_STRING {
                  STRSIZE H5T_VARIABLE;
                  STRPAD H5T_STR_NULLTERM;
                  CSET H5T_CSET_ASCII;
                  CTYPE H5T_C_S1;
               }
               DATASPACE  SIMPLE { ( 1 ) / ( 1 ) }
            }
            ATTRIBUTE "units" {
               DATATYPE  H5T_STRING {
                  STRSIZE H5T_VARIABLE;
                  STRPAD H5T_STR_NULLTERM;
                  CSET H5T_CSET_ASCII;
                  CTYPE H5T_C_S1;
               }
               DATASPACE  SIMPLE { ( 1 ) / ( 1 ) }
            }
         }

Any ideas how to write the units as '[deg]' which the C comparator will accept?

Or is it the if (ok) { line just above which is not getting run? I need some help with debugging since I cant debug tomviz directly.

[cryos]

We read and write these in the external pipeline code, see here for the write routine. I can dig into this a little more, there may be some bugs remaining in the IO code that we missed.

[ercius]

I did not create the softlink for the active scalars. I dont think this will affect me though.

I see this _read_emd function uses a different way to indicate a tilt series. See here. It looks for b'angles' or 'units' in ANGLE_UNITS

Ill try that.

[ercius]

I tried to match this correctly, but it still does not work. if dims[0].name == b'angles'

There must be something else that Im missing to trigger it to recognize it as a tilt series. Or the string comparison is wrong for some strange reason.

[psavery]

Hi @ercius. I gave this a try, and setting the attribute via:

fid['/data/tomography/dim1'].attrs['units'] = "[deg]"

Seemed to work for me. Note that I also had to make sure the data type of the angles was float32 (since Tomviz expects this dtype here), although it looks to me that you did that as well.

I'm gonna paste some example code I tried to see if it will be helpful.

import h5py
import numpy as np

filepath = "out.emd"

# Make up some data
shape = (74, 256, 256)
array = np.arange(np.prod(shape), dtype=np.float32).reshape(shape)

with h5py.File(filepath, "w") as f:
    group = f.create_group("/data/tomography")
    group["data"] = array

    # Tilt angles
    # MUST be float32 (tomviz is expecting this type)
    group["dim1"] = np.arange(-75, 77, 2, dtype=np.float32)
    group["dim1"].attrs["name"] = "angles"
    group["dim1"].attrs["units"] = "[deg]"

    # y
    # MUST be float32 (tomviz is expecting this type)
    group["dim2"] = np.arange(shape[1], dtype=np.float32) * 2
    group["dim2"].attrs["name"] = "y"
    group["dim2"].attrs["units"] = "[n_m]"

    # x
    # MUST be float32 (tomviz is expecting this type)
    group["dim3"] = np.arange(shape[2], dtype=np.float32) + 5
    group["dim3"].attrs["name"] = "x"
    group["dim3"].attrs["units"] = "[n_m]"

If you load this into Tomviz, and then click the dataset name in the top left corner, and scroll down on the data properties panel (located in the bottom left corner), you can see the tilt angles at the bottom. That is one way to see if Tomviz has recognized the dataset as a tilt series.

Note that Tomviz will automatically swap the first and last axes of the dataset when it recognizes it as a tilt series as well.

Here's the h5dump for that dimension:

DATASET "/data/tomography/dim1" {
   DATATYPE  H5T_IEEE_F32LE
   DATASPACE  SIMPLE { ( 76 ) / ( 76 ) }
   DATA {
   (0): -75, -73, -71, -69, -67, -65, -63, -61, -59, -57, -55, -53, -51, -49,
   (14): -47, -45, -43, -41, -39, -37, -35, -33, -31, -29, -27, -25, -23,
   (27): -21, -19, -17, -15, -13, -11, -9, -7, -5, -3, -1, 1, 3, 5, 7, 9, 11,
   (44): 13, 15, 17, 19, 21, 23, 25, 27, 29, 31, 33, 35, 37, 39, 41, 43, 45,
   (61): 47, 49, 51, 53, 55, 57, 59, 61, 63, 65, 67, 69, 71, 73, 75
   }
   ATTRIBUTE "name" {
      DATATYPE  H5T_STRING {
         STRSIZE H5T_VARIABLE;
         STRPAD H5T_STR_NULLTERM;
         CSET H5T_CSET_UTF8;
         CTYPE H5T_C_S1;
      }
      DATASPACE  SCALAR
      DATA {
      (0): "angles"
      }
   }
   ATTRIBUTE "units" {
      DATATYPE  H5T_STRING {
         STRSIZE H5T_VARIABLE;
         STRPAD H5T_STR_NULLTERM;
         CSET H5T_CSET_UTF8;
         CTYPE H5T_C_S1;
      }
      DATASPACE  SCALAR
      DATA {
      (0): "[deg]"
      }
   }
}

[ercius]

Great! Ill give this a try with your example code. This will make it easier to work with data in and out of Tomviz.

[ercius]

This is working for me now as well. Thanks!