ENH: Read RAPIC files

[jjhelmus]

Py-ART could support reading of RAPIC files.

Currently the suggestion is to use Radx to read and convert these files to Cf/Radial NetCDF files which can be read in by Py-ART using pyart.io.read.

Sometimes multiple volumes are combined into a single files. These files can be split using the following:

import re

with open('filename.rapic', 'rb') as f:
    buf = f.read()

splits = [m.start() for m in re.finditer('/IMAGE:', buf)]
splits.append(len(buf))

for split_num in range(len(splits) - 1):
    start = splits[split_num]
    end = splits[split_num + 1]
    fname = 'rapic_split.' + str(split_num + 1)
    print "Writing:", fname
    with open(fname, 'wb') as out:
        out.write(buf[start:end])

[joshua-wx]

Just a note: The rapic format at the BOM is planned to be made redundant sometime in the future and replaced with ohimh5. We now have an excellent in house utility for converting streamed rapic data to odimh5 and all new science projects are using this format. The latest radar monitoring/archive project (rain fields) also uses the odimh5 format.

[scollis]

@joshuass : Does BoM plan to opensource. We have 10 years of legacy data from Berrimah and other radars.

[joshua-wx]

Yes. I spoke to the dev today and he'll put in a request with management to release the conversion utility as open source and host it on github. We should get an answer quite quickly too. I'll let you know!

[jjhelmus]

Really exciting news @joshuass, will be hoping that this can get release.

In either case it sounds like adding RAPIC support to Py-ART is a low priority. I want to keep this PR open but I'll tag to indicate that it will not be a development priority

[joshua-wx]

I can confirm the utility source code will be up on an Australian BOM github account within a week :smile: . I've tested it on 18 years worth of radar data and across a majority of the network with no issues, it's very fast too (written in C++). The only limitation is that it cannot process RHI RAPIC data, but only research radars collect RHIs and this data is available in mdv.

[joshua-wx]

An update from the BOM radar dev, Mark Curtis. Link to the rapic_to_odim converter source on github is below too!

Hi Josh,

Our convenience libraries for working with rapic and ODIM format files (including our rapic to ODIM converter) are now open sourced under the Apache 2.0 license and available on GitHub. They live at the following location:

http://github.com/bom-radar

To compile the rapic_to_odim utility, you will need to install both libraries. The odim_h5 library should be installed first. The rapic library will then detect the presence of odim_h5 during build and compile the rapic_to_odim binary.

As I mentioned to you earlier the rapic converter currently only targets volume scans, since that is all I've needed for my purposes so far. I'm always willing to accept patches to support RHI sets from rapic - it shouldn't be hard to do really as the rapic parsing itself is common code. Also, the converter assumes you have one volume per file, so support for concatenated rapic files (so called 'rapic databases') is missing. This would also be quite easy to add but I have not had a need thus far.

Both libraries are written in C++11, meaning you will likely need to add a -std=c++11 or better flag to your compile line for anything that uses the libraries directly.

If you run into any technical problems just open an issue on the relevant GitHub repo. I should be able to respond to them fairly quickly.

Please feel free to distribute the URL and forward this email as you see fit.

Regards,

Mark

[jjhelmus]

Awesome, glad to see this available for the community. Thanks for your work on getting this out @joshuass!