ACT
ACT copied to clipboard
ARM-DOE
Xarray not able to read old MMCR data
Some older data might not be compatible with xarray. sgpmmcrmomC1.b1.20041107 data yielded this error:
xarray.core.variable.MissingDimensionsError: 'heights' has more than 1-dimension and the same name as one of its dimensions ('mode', 'heights'). xarray disallows such variables because they conflict with the coordinates used to label dimensions.
Is there a way around this?
You can specify decode_coords=False to xarray.open_dataset and that should at least disable the dimension labelling for nasty datasets like this.
From: Adam Theisen [email protected] Sent: Wednesday, May 22, 2019 9:33:25 AM To: ANL-DIGR/ACT Cc: Subscribed Subject: [ANL-DIGR/ACT] Xarray not able to read old MMCR data (#67)
Some older data might not be compatible with xarray. sgpmmcrmomC1.b1.20041107 data yielded this error:
xarray.core.variable.MissingDimensionsError: 'heights' has more than 1-dimension and the same name as one of its dimensions ('mode', 'heights'). xarray disallows such variables because they conflict with the coordinates used to label dimensions.
Is there a way around this?
— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHubhttps://github.com/ANL-DIGR/ACT/issues/67?email_source=notifications&email_token=AFIQA5DSOSEQ26RJ2ERS3M3PWVKTLA5CNFSM4HOU6TA2YY3PNVWWK3TUL52HS4DFUVEXG43VMWVGG33NNVSW45C7NFSM4GVHKM6Q, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AFIQA5FHMG6NXEAFKAGDYNLPWVKTLANCNFSM4HOU6TAQ.
netcdf sgpmmcrmomC1.b1.20041107.000014 { dimensions: time = UNLIMITED ; // (44959 currently) mode = 10 ; namelength = 64 ; heights = 167 ; hourly = 24 ; variables: int base_time ; base_time:string = "7-Nov-2004,0:00:14 GMT" ; base_time:long_name = "Base time in Epoch" ; base_time:units = "seconds since 1970-1-1 0:00:00 0:00" ; double time_offset(time) ; time_offset:long_name = "Time offset from base_time" ; time_offset:units = "seconds since 2004-11-07 00:00:14 0:00" ; double time(time) ; time:long_name = "Time offset from midnight" ; time:units = "seconds since 2004-11-07 00:00:00 0:00" ;
float heights(mode, heights) ;
heights:long_name = "Range Heights (center of radar sample volume)" ;
heights:units = "m MSL" ;
heights:missing_value = -9999.f ;
int DataQualityStatus(time) ;
DataQualityStatus:long_name = "Data Quality Status" ;
DataQualityStatus:units = "code" ;
DataQualityStatus:missing_value = -9999 ;
float MeanDopplerVelocity(time, heights) ;
MeanDopplerVelocity:long_name = "Mean Doppler Velocity" ;
MeanDopplerVelocity:units = "m/s" ;
MeanDopplerVelocity:resolution = 0.001f ;
MeanDopplerVelocity:missing_value = -9999.f ;
float SignalToNoiseRatio(time, heights) ;
SignalToNoiseRatio:long_name = "Signal to Noise Ratio" ;
SignalToNoiseRatio:units = "dB" ;
SignalToNoiseRatio:resolution = 0.001f ;
SignalToNoiseRatio:missing_value = -9999.f ;
float Power(time, heights) ;
Power:long_name = "Power (uncalibrated)" ;
Power:units = "dB" ;
Power:resolution = 0.001f ;
Power:missing_value = -9999.f ;
float SpectralWidth(time, heights) ;
SpectralWidth:long_name = "Spectral Width" ;
SpectralWidth:units = "m/s" ;
SpectralWidth:resolution = 0.001f ;
SpectralWidth:missing_value = -9999.f ;
float NoiseLevel(time, heights) ;
NoiseLevel:long_name = "Mean Noise Level" ;
NoiseLevel:units = "dB" ;
NoiseLevel:resolution = 0.001f ;
NoiseLevel:missing_value = -99999.f ;
float Reflectivity(time, heights) ;
Reflectivity:long_name = "Reflectivity" ;
Reflectivity:units = "dBZ" ;
Reflectivity:resolution = 0.001f ;
Reflectivity:missing_value = -9999.f ;
Reflectivity:comment1 = "Nearfield correction has been applied" ;
Reflectivity:comment2 = "Reference: Sekelsky, S. (2002), JAOT, v19, no. 4" ;
float RangeCorrectedPower(time, heights) ;
RangeCorrectedPower:long_name = "Range Corrected Calibrated Power" ;
RangeCorrectedPower:units = "dBm" ;
RangeCorrectedPower:resolution = 0.001f ;
RangeCorrectedPower:missing_value = -9999.f ;
float CircularDepolarizationRatio(time, heights) ;
CircularDepolarizationRatio:long_name = "Circular Depolarization Ratio" ;
CircularDepolarizationRatio:units = "dB" ;
CircularDepolarizationRatio:resolution = 0.001f ;
CircularDepolarizationRatio:missing_value = -9999.f ;
Then we would need a procedure that looks for variables that have the same name as a dimension in a netCDF file. Then it would suggest a change to the user and save a new file for xarray compatibility.
Bobby
From: Adam Theisen [email protected] Sent: Wednesday, May 22, 2019 9:45:31 AM To: ANL-DIGR/ACT Cc: Jackson, Robert; Comment Subject: Re: [ANL-DIGR/ACT] Xarray not able to read old MMCR data (#67)
netcdf sgpmmcrmomC1.b1.20041107.000014 { dimensions: time = UNLIMITED ; // (44959 currently) mode = 10 ; namelength = 64 ; heights = 167 ; hourly = 24 ; variables: int base_time ; base_time:string = "7-Nov-2004,0:00:14 GMT" ; base_time:long_name = "Base time in Epoch" ; base_time:units = "seconds since 1970-1-1 0:00:00 0:00" ; double time_offset(time) ; time_offset:long_name = "Time offset from base_time" ; time_offset:units = "seconds since 2004-11-07 00:00:14 0:00" ; double time(time) ; time:long_name = "Time offset from midnight" ; time:units = "seconds since 2004-11-07 00:00:00 0:00" ;
float heights(mode, heights) ;
heights:long_name = "Range Heights (center of radar sample volume)" ;
heights:units = "m MSL" ;
heights:missing_value = -9999.f ;
int DataQualityStatus(time) ;
DataQualityStatus:long_name = "Data Quality Status" ;
DataQualityStatus:units = "code" ;
DataQualityStatus:missing_value = -9999 ;
float MeanDopplerVelocity(time, heights) ;
MeanDopplerVelocity:long_name = "Mean Doppler Velocity" ;
MeanDopplerVelocity:units = "m/s" ;
MeanDopplerVelocity:resolution = 0.001f ;
MeanDopplerVelocity:missing_value = -9999.f ;
float SignalToNoiseRatio(time, heights) ;
SignalToNoiseRatio:long_name = "Signal to Noise Ratio" ;
SignalToNoiseRatio:units = "dB" ;
SignalToNoiseRatio:resolution = 0.001f ;
SignalToNoiseRatio:missing_value = -9999.f ;
float Power(time, heights) ;
Power:long_name = "Power (uncalibrated)" ;
Power:units = "dB" ;
Power:resolution = 0.001f ;
Power:missing_value = -9999.f ;
float SpectralWidth(time, heights) ;
SpectralWidth:long_name = "Spectral Width" ;
SpectralWidth:units = "m/s" ;
SpectralWidth:resolution = 0.001f ;
SpectralWidth:missing_value = -9999.f ;
float NoiseLevel(time, heights) ;
NoiseLevel:long_name = "Mean Noise Level" ;
NoiseLevel:units = "dB" ;
NoiseLevel:resolution = 0.001f ;
NoiseLevel:missing_value = -99999.f ;
float Reflectivity(time, heights) ;
Reflectivity:long_name = "Reflectivity" ;
Reflectivity:units = "dBZ" ;
Reflectivity:resolution = 0.001f ;
Reflectivity:missing_value = -9999.f ;
Reflectivity:comment1 = "Nearfield correction has been applied" ;
Reflectivity:comment2 = "Reference: Sekelsky, S. (2002), JAOT, v19, no. 4" ;
float RangeCorrectedPower(time, heights) ;
RangeCorrectedPower:long_name = "Range Corrected Calibrated Power" ;
RangeCorrectedPower:units = "dBm" ;
RangeCorrectedPower:resolution = 0.001f ;
RangeCorrectedPower:missing_value = -9999.f ;
float CircularDepolarizationRatio(time, heights) ;
CircularDepolarizationRatio:long_name = "Circular Depolarization Ratio" ;
CircularDepolarizationRatio:units = "dB" ;
CircularDepolarizationRatio:resolution = 0.001f ;
CircularDepolarizationRatio:missing_value = -9999.f ;
— You are receiving this because you commented. Reply to this email directly, view it on GitHubhttps://github.com/ANL-DIGR/ACT/issues/67?email_source=notifications&email_token=AFIQA5FQYYMDPA7MAQTAHYDPWVMAXA5CNFSM4HOU6TA2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODV7JHIA#issuecomment-494834592, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AFIQA5GMZQJFE2RADGDTQZDPWVMAXANCNFSM4HOU6TAQ.
Oh crud.. it puts all the modes in one file!
Bobby is right.. we effectively need a mode splitter
From: Bobby Jackson [email protected] Reply-To: ANL-DIGR/ACT [email protected] Date: Wednesday, May 22, 2019 at 9:51 AM To: ANL-DIGR/ACT [email protected] Cc: Scott [email protected], Comment [email protected] Subject: Re: [ANL-DIGR/ACT] Xarray not able to read old MMCR data (#67)
Then we would need a procedure that looks for variables that have the same name as a dimension in a netCDF file. Then it would suggest a change to the user and save a new file for xarray compatibility.
Bobby
From: Adam Theisen [email protected] Sent: Wednesday, May 22, 2019 9:45:31 AM To: ANL-DIGR/ACT Cc: Jackson, Robert; Comment Subject: Re: [ANL-DIGR/ACT] Xarray not able to read old MMCR data (#67)
netcdf sgpmmcrmomC1.b1.20041107.000014 { dimensions: time = UNLIMITED ; // (44959 currently) mode = 10 ; namelength = 64 ; heights = 167 ; hourly = 24 ; variables: int base_time ; base_time:string = "7-Nov-2004,0:00:14 GMT" ; base_time:long_name = "Base time in Epoch" ; base_time:units = "seconds since 1970-1-1 0:00:00 0:00" ; double time_offset(time) ; time_offset:long_name = "Time offset from base_time" ; time_offset:units = "seconds since 2004-11-07 00:00:14 0:00" ; double time(time) ; time:long_name = "Time offset from midnight" ; time:units = "seconds since 2004-11-07 00:00:00 0:00" ;
float heights(mode, heights) ; heights:long_name = "Range Heights (center of radar sample volume)" ; heights:units = "m MSL" ; heights:missing_value = -9999.f ; int DataQualityStatus(time) ; DataQualityStatus:long_name = "Data Quality Status" ; DataQualityStatus:units = "code" ; DataQualityStatus:missing_value = -9999 ; float MeanDopplerVelocity(time, heights) ; MeanDopplerVelocity:long_name = "Mean Doppler Velocity" ; MeanDopplerVelocity:units = "m/s" ; MeanDopplerVelocity:resolution = 0.001f ; MeanDopplerVelocity:missing_value = -9999.f ; float SignalToNoiseRatio(time, heights) ; SignalToNoiseRatio:long_name = "Signal to Noise Ratio" ; SignalToNoiseRatio:units = "dB" ; SignalToNoiseRatio:resolution = 0.001f ; SignalToNoiseRatio:missing_value = -9999.f ; float Power(time, heights) ; Power:long_name = "Power (uncalibrated)" ; Power:units = "dB" ; Power:resolution = 0.001f ; Power:missing_value = -9999.f ; float SpectralWidth(time, heights) ; SpectralWidth:long_name = "Spectral Width" ; SpectralWidth:units = "m/s" ; SpectralWidth:resolution = 0.001f ; SpectralWidth:missing_value = -9999.f ; float NoiseLevel(time, heights) ; NoiseLevel:long_name = "Mean Noise Level" ; NoiseLevel:units = "dB" ; NoiseLevel:resolution = 0.001f ; NoiseLevel:missing_value = -99999.f ; float Reflectivity(time, heights) ; Reflectivity:long_name = "Reflectivity" ; Reflectivity:units = "dBZ" ; Reflectivity:resolution = 0.001f ; Reflectivity:missing_value = -9999.f ; Reflectivity:comment1 = "Nearfield correction has been applied" ; Reflectivity:comment2 = "Reference: Sekelsky, S. (2002), JAOT, v19, no. 4" ; float RangeCorrectedPower(time, heights) ; RangeCorrectedPower:long_name = "Range Corrected Calibrated Power" ; RangeCorrectedPower:units = "dBm" ; RangeCorrectedPower:resolution = 0.001f ; RangeCorrectedPower:missing_value = -9999.f ; float CircularDepolarizationRatio(time, heights) ; CircularDepolarizationRatio:long_name = "Circular Depolarization Ratio" ; CircularDepolarizationRatio:units = "dB" ; CircularDepolarizationRatio:resolution = 0.001f ; CircularDepolarizationRatio:missing_value = -9999.f ;
— You are receiving this because you commented. Reply to this email directly, view it on GitHubhttps://github.com/ANL-DIGR/ACT/issues/67?email_source=notifications&email_token=AFIQA5FQYYMDPA7MAQTAHYDPWVMAXA5CNFSM4HOU6TA2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODV7JHIA#issuecomment-494834592, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AFIQA5GMZQJFE2RADGDTQZDPWVMAXANCNFSM4HOU6TAQ.
— You are receiving this because you commented. Reply to this email directly, view it on GitHub, or mute the thread.
Yep, this was the old fashion way of handling modes. Interleaved in the data. Mode varied with time and you had to pull out the modes. Fun times
I think I have a method down here that uses the NetCDF4 library. The only question then is if we want to expand the function to breakout the modes.