ARM-DOE
ENH: Read new variation of hdf5 file format
- Py-ART version: 1.17.0 py39ha55989b_0
- Python version: 3.9.18 h1aa4202_0
- Operating System: windows 11
Description
Hello, Im trying to read in some hdf5 data obtained from one brazilian radar. Pyart doesn't seem to be able to read in the file, h5py can. I have tried gamin_hdf5, and posted the error message below at the end after my code.
If it helps I have shared the data (found below) https://drive.google.com/file/d/1RLOyF2tkuFDd1BbmTUPLiei6mIEkW4m1/view?usp=sharing
Thanks for any advice explanation on why this doesn't work.
What I Did
import pyart
import matplotlib.pyplot as plt
import numpy as np
import cartopy.crs as ccrs
import h5py
radar = pyart.aux_io.read_gamic('radarchapeco.hdf5')
---------------------------------------------------------------------------
TypeError Traceback (most recent call last)
Cell In[3], line 1
----> 1 radar = pyart.aux_io.read_gamic('radarchapeco.hdf5')
File ~\miniconda3\lib\site-packages\pyart\aux_io\gamic_hdf5.py:107, in read_gamic(filename, field_names, additional_metadata, file_field_names, exclude_fields, include_fields, valid_range_from_file, units_from_file, pulse_width, **kwargs)
97 filemetadata = FileMetadata(
98 "gamic",
99 field_names,
(...)
103 include_fields,
104 )
106 # Open HDF5 file and get handle
--> 107 gfile = GAMICFile(filename)
109 # verify that all scans are present in file
110 assert gfile.is_file_complete()
File ~\miniconda3\lib\site-packages\pyart\aux_io\gamicfile.py:40, in GAMICFile.__init__(self, filename)
38 self._hfile = h5py.File(filename, "r")
39 self.nsweeps = self._hfile["what"].attrs["sets"]
---> 40 self._scans = ["scan%i" % (i) for i in range(self.nsweeps)]
41 self.rays_per_sweep = self.how_attrs("ray_count", "int32")
42 self.total_rays = sum(self.rays_per_sweep)
TypeError: only integer scalar arrays can be converted to a scalar index
Hello @megumillia ! I can take a look real quick and see. My guess is the attribute nsweep is not an integer
@megumillia The issue i'm seeing is alot of the attributes in this file are in arrays, instead of strings, int etc. For example your nsweeps instead of 11 is np.ndarray([11]) scan_type is ['ppi'] instead of ppi, etc which py-art does not like as its not expecting arrays for these values
That's not the only issue here. The timestamps are missing from the ray_header.
@kmuehlbauer Exactly!
Funny enough, was just dissecting that part of the file now haha: hfile["scan1"]["ray_header"].attrs.keys() <KeysViewHDF5 []>
I can think of ways for checking for np.ndarray for the arrays of some of the attributes, but is there anyway around the missing timestamps @kmuehlbauer ? Not too sure how to approach this.
We would need to read the sweep timestamps and try to derive ray times from rotation speed.
@megumillia I can try a few things and see if I can get it to read.
if you can i will be gratefull, I'm new to this programming and radar data thing.
@megumillia At the moment I have ways around the np.ndarray parts, but the missing sweep timestamps as @kmuehlbauer mentioned, would require deriving ray times from rotation sweep. That I'm unfamiliar with sadly
@megumillia - a new reader would likely need to be written here. If you have the data read in using h5py, I encourage you to check out our notebook example fitting data into the radar object https://arm-doe.github.io/pyart/notebooks/basic_ingest_using_test_radar_object.html
This would be the best path forward, and once you have a working example, we could work together on a pull request to Py-ART or xradar https://github.com/openradar/xradar (which might be the better option) to have this fully supported. As of now, the file does not conform to any of the readers implemented in Py-ART or xradar.
@mgrover1 Thanks for the tip and the example link, I'm working on it.
When i finish, i can post here the final code if you want do a pull request to Py-art or xradar.