Pollynet_Processing_Chain
Pollynet_Processing_Chain copied to clipboard
PollyNET
store configuration file in netcdf
To allow reproducibility, I would propose to store the whole config file which has been used, i.e. all parameters, in the netcdf as well. Maybe as simple character variable first. But this would help to identify issues in case the processing was made elsewhere....
I tried to do so.
- I just wanted to dump the whole PollyConfig global variable in one string and store it as one global attribute in the netcdf file. Similar to the history global attribute in each netcdf: :history = "Last processing time at 2024-07-31 13:49:40 by pollySaveProfiles, git branch: cleaning_up, git commit: 6...";
However I failed to convert the whole PollyConfig struct to a string.
because matlab cannot easily do this for type 'struct'
- I tried to use a for loop reading the field names of struct PollyConfig, and put this as attribute together with the respective value: names=fieldnames(PollyConfig) values = struct2cell(PollyConfig) s = strings for i=1:size(names) s = strcat(s,names,string(values)) end however I failed because the respective data type of the fields is always different:
Any idea/advise? @ZPYin @cristoferjimenez @ulysses78 @Moritz-TROPOS @AthenaAugousta
Best Holger
Hi Holger,
I think in that case, one needs to adapt the types of each parameter. I tried this change and it works.
Best regards, Cristofer
s = strings
for i=1:size(names)
type= class(values{i});
if strcmp(type,'cell')
values2=num2str(str2double(values{i})) ;
elseif strcmp(type,'logical')
values2=num2str(double(values{i}));
else
values2=(values{i});
if strcmp(type,'double')
if size(values2,1)>1
values2=num2str(reshape(values2,1,size(values2,1)*size(values2,2)));
else
values2=num2str(values2);
end
end
end
s = strcat(s,names{i},',',string(values2),',')
end
Hi Holger,
If you want to convert a Matlab struct to a char array or a string, you need to specify the output format for different data types (as suggested by @cristoferjimenez ).
I modified the code (from @cristoferjimenez ) a little bit to make it support the conversion of PollyConfig struct. Please see the two little functions in the link below.
matlab-funcs-struct-2-char.zip
Below is an output test.
PollyConfig
{
"flagCorrectFalseMShots": true,
"flagFilterFalseMShots": false,
"flagDTCor": true,
"flagWVCalibration": true,
"flagUsePreviousDepolCali": true,
"flagUsePreviousWVconst": true,
"flagUsePreviousLC": true,
"dataFileFormat": "(?<year>\\d{4})_(?<month>\\d{2})_(?<day>\\d{2})_\\w*_(?<hour>\\d{2})_(?<minute>\\d{2})_(?<second>\\d{2})\\w*.nc",
"gdas1Site": "cabauw",
"AERONETSite": "Cabauw",
"max_height_bin": 3000,
"first_range_gate_indx": [261, 261, 261, 261, 261, 261, 261, 261, 262, 262],
"first_range_gate_height": 78.75,
"dtCorMode": 2,
"dt": [[0.0, 0.972992, 0.00353332, -7.90981e-006, 1.06451e-007, 1.42895e-009],
[0, 1.0117, -0.0014, 0.0002, -0.0000, 0.0000],
[0, 0.9674, 0.0023, 0.0000, 0.0000, 0.0000],
[0, 0.9929, 0.0000, 0.0001, -0.0000, 0.0000],
[0, 0.9843, 0.0022, 0.0001, -0.0000, 0.0000],
[0, 0.9391, 0.0063, -0.0001, 0.0000, -0.0000],
[0, 1.0035, 0.0003, 0.0001, -0.0000, 0.0000],
[0, 1.0000, 0, 0, 0, 0],
[0, 1.0000, 0.0029, 0.0000, 0.0000, 0.0000],
[0, 1.0000, 0.0028, 0.0000, 0.0000, 0.0000]],
"bgCorRangeIndx": [10, 240],
"mask_SNRmin": [1, 1, 1, 0.01, 1, 1, 1, 1, 1, 1],
"init_depAng": 19.99,
"maskDepCalAng": ["none", "none", "p", "p", "p", "p", "p", "p", "p", "p", "none", "none", "n", "n", "n", "n", "n", "n", "n", "n", "none"],
"depol_cal_minbin_532": 40,
"depol_cal_maxbin_532": 300,
"depol_cal_SNRmin_532": [1, 1, 1, 1],
"depol_cal_sigMax_532": [1500, 1500, 1500, 1500],
"rel_std_dplus_532": 0.2,
"rel_std_dminus_532": 0.2,
"depol_cal_segmentLen_532": 40,
"depol_cal_smoothWin_532": 8,
"depol_cal_minbin_355": 40,
"depol_cal_maxbin_355": 300,
"depol_cal_SNRmin_355": [2, 2, 2, 2],
"depol_cal_sigMax_355": [1000, 1000, 1000, 1000],
"rel_std_dplus_355": 0.2,
"rel_std_dminus_355": 0.2,
"depol_cal_segmentLen_355": 40,
"depol_cal_smoothWin_355": 8,
"isFR": [1, 1, 1, 1, 1, 1, 1, 1, 0, 0],
"isNR": [0, 0, 0, 0, 0, 0, 0, 0, 1, 1],
"is532nm": [0, 0, 0, 0, 1, 1, 0, 0, 1, 0],
"is355nm": [1, 1, 0, 0, 0, 0, 0, 0, 0, 0],
"is1064nm": [0, 0, 0, 0, 0, 0, 0, 1, 0, 0],
"isTot": [1, 0, 0, 0, 1, 0, 0, 1, 1, 0],
"isCross": [0, 1, 0, 0, 0, 1, 0, 0, 0, 0],
"is387nm": [0, 0, 1, 0, 0, 0, 0, 0, 0, 0],
"is407nm": [0, 0, 0, 1, 0, 0, 0, 0, 0, 0],
"is607nm": [0, 0, 0, 0, 0, 0, 1, 0, 0, 1],
"channelTag": ["FR-total-355 nm", "FR-cross-355 nm", "FR-387 nm", "FR-407 nm", "FR-total-532 nm", "FR-cross-532 nm", "FR-607 nm", "FR-total-1064 nm", "NR-total-532 nm", "NR-607 nm"],
"minPC_fog": 60,
"TR": [0.69, 500, 1, 1, 1.09, 500, 1, 1, 1, 1],
"overlapCalMode": 1,
"maxSigSlope4FilterCloud": 3e6,
"saturate_thresh": 100,
"heightFullOverlap": [500, 500, 500, 500, 500, 500, 500, 500, 150, 150],
"minSNR_4_sigNorm": [10],
"intNProfiles": 120,
"minIntNProfiles": 90,
"meteorDataSource": "gdas1",
"radiosondeSitenum": 0,
"IWV_instrument": "AERONET",
"maxIWVTLag": 0.0833,
"minDecomLogDist355": 0.2,
"minDecomLogDist532": 0.2,
"minDecomLogDist1064": 0.2,
"maxDecomHeight355": 8000,
"maxDecomHeight532": 8000,
"maxDecomHeight1064": 8000,
"maxDecomThickness355": 700,
"maxDecomThickness532": 700,
"maxDecomThickness1064": 700,
"decomSmoothWin355": 60,
"decomSmoothWin532": 60,
"decomSmoothWin1064": 120,
"minRefThickness355": 300,
"minRefThickness532": 300,
"minRefThickness1064": 300,
"minRefDeltaExt355": 1,
"minRefDeltaExt532": 1,
"minRefDeltaExt1064": 1,
"minRefSNR355": 5,
"minRefSNR532": 5,
"minRefSNR1064": 4,
"LR355": 50,
"LR532": 50,
"LR1064": 50,
"refBeta355": 2e-8,
"refBeta532": 1e-8,
"refBeta1064": 2e-9,
"smoothWin_klett_355": 21,
"smoothWin_klett_532": 21,
"smoothWin_klett_1064": 21,
"maxIterConstrainFernald": 20,
"minLRConstrainFernald": 1,
"maxLRConstrainFernald": 150,
"minDeltaAOD": 0.01,
"minRamanRefSNR387": 20,
"minRamanRefSNR607": 10,
"angstrexp": 0.9,
"smoothWin_raman_355": 61,
"smoothWin_raman_532": 61,
"smoothWin_raman_1064": 61,
"LCMeanWindow": 50,
"LCMeanMinIndx": 70,
"LCMeanMaxIndx": 1000,
"LCCalibrationStatus": ["none", "Klett", "Raman", "Defaults", "History"],
"quasi_smooth_h": [8, 8, 8, 8, 8, 8, 8, 8, 8, 8],
"quasi_smooth_t": [10, 10, 10, 10, 10, 10, 10, 10, 10, 10],
"hWVCaliBase": 120,
"hWVCaliTop": 8000,
"minSNRWVCali": 10,
"clear_thres_par_beta_1064": 1e-8,
"turbid_thres_par_beta_1064": 2e-7,
"turbid_thres_par_beta_532": 2e-7,
"droplet_thres_par_depol": 0.05,
"spheroid_thres_par_depol": 0.07,
"unspheroid_thres_par_depol": 0.2,
"ice_thres_par_depol": 0.35,
"ice_thres_vol_depol": 0.3,
"large_thres_ang": 0.75,
"small_thres_ang": 0.5,
"cloud_thres_par_beta_1064": 2e-5,
"min_atten_par_beta_1064": 10,
"search_cloud_above": 300,
"search_cloud_below": 100,
"overlap532Color": [0, 255, 64],
"overlap355Color": [0, 0, 255],
"xLim_Profi_RCS": [0.3, 10],
"xLim_Profi_Bsc": [-0.05, 3],
"xLim_Profi_Ext": [-1, 150],
"xLim_Profi_LR": [0, 120],
"xLim_Profi_WV_RH": [0, 10],
"xLim_Profi_WV_RH": [0, 8],
"yLim_FR_RCS": [0, 20000],
"yLim_NR_RCS": [0, 3000],
"yLim_LC_355": [0, 1e14],
"yLim_LC_532": [0, 2e14],
"yLim_LC_1064": [0, 6e14],
"yLim_LC_387": [0, 1e14],
"yLim_LC_607": [0, 3e14],
"yLim_WVConst": [0, 20],
"yLim_depolConst_355": [0, 0.2],
"yLim_depolConst_532": [0, 0.2],
"zLim_att_beta_355": [0, 15],
"zLim_att_beta_532": [0, 5],
"zLim_att_beta_1064": [0, 2],
"zLim_quasi_beta_532": [0, 2],
"zLim_quasi_beta_1064": [0, 0.8],
"zLim_quasi_Par_DR_532": [0, 0.4],
"zLim_FR_RCS_355": [1e-2, 15],
"zLim_FR_RCS_532": [1e-2, 15],
"zLim_FR_RCS_1064": [1e-2, 15],
"zLim_NR_RCS_355": [1e-2, 2],
"zLim_NR_RCS_532": [1e-2, 2],
"calibrationDB": "arielle_calibration.db",
"logbookFile": "/data/level1a/polly/logbook_polly_Arielle_til_2018.csv"
}
Output
flagCorrectFalseMShots,1,flagFilterFalseMShots,0,flagForceMeasTime,0,flagDTCor,1,flagSigTempCor,0,flagLCCalibration,1,flagDepolCali,1,flagWVCalibration,1,flagUsePreviousDepolCali,1,flagUsePreviousWVconst,1,flagUsePreviousLC,1,flagUseManualRefH,0,flagUseSameRefH,0,flagAutoscaleRCS,1,flagMolDepolCali,0,flagTransCor,1,MWRFolder,,dataFileFormat,(?<year>\d{4})_(?<month>\d{2})_(?<day>\d{2})_\w*_(?<hour>\d{2})_(?<minute>\d{2})_(?<second>\d{2})\w*.nc,gdas1Site,cabauw,AERONETSite,Cabauw,max_height_bin,3000,first_range_gate_indx,261 261 261 261 261 261 261 261 262 262,first_range_gate_height,78.75,deltaT,30,dtCorModeLabel,[1],Polynomial coefficients stores in netcdf file,[2],nonparalyzable correction,[3],User defined polynomial cofficients,,dtCorMode,2,dt,0 0 0 0 0 0 0 0 0 0 0.97299 1.0117 0.9674 0.9929 0.9843 0.9391 1.0035 1 1 1 0.0035333 -0.0014 0.0023 0 0.0022 0.0063 0.0003 0 0.0029 0.0028 -7.9098e-06 0.0002 0 0.0001 0.0001 -0.0001 0.0001 0 0 0 1.0645e-07 0 0 0 0 0 0 0 0 0 1.4289e-09 0 0 0 0 0 0 0 0 0,bgCorRangeIndx,10 240,mask_SNRmin,1 1 1 0.01 1 1 1 1 1 1,tempCorFunc,[1],1,[2],1,[3],1,[4],1,[5],1,[6],1,[7],1,[8],1,[9],1,[10],1,[11],1,[12],1,[13],1,,init_depAng,19.99,depol_cali_mode,2,depol_cal_time_fixed_p_start,,depol_cal_time_fixed_p_end,,depol_cal_time_fixed_m_start,,depol_cal_time_fixed_m_end,,maskDepCalAng,[1],none,[2],none,[3],p,[4],p,[5],p,[6],p,[7],p,[8],p,[9],p,[10],p,[11],none,[12],none,[13],n,[14],n,[15],n,[16],n,[17],n,[18],n,[19],n,[20],n,[21],none,,depol_cal_minbin_532,40,depol_cal_maxbin_532,300,depol_cal_SNRmin_532,1 1 1 1,depol_cal_sigMax_532,1500 1500 1500 1500,rel_std_dplus_532,0.2,rel_std_dminus_532,0.2,depol_cal_segmentLen_532,40,depol_cal_smoothWin_532,8,depol_cal_minbin_355,40,depol_cal_maxbin_355,300,depol_cal_SNRmin_355,2 2 2 2,depol_cal_sigMax_355,1000 1000 1000 1000,rel_std_dplus_355,0.2,rel_std_dminus_355,0.2,depol_cal_segmentLen_355,40,depol_cal_smoothWin_355,8,depol_cal_minbin_1064,100,depol_cal_maxbin_1064,300,depol_cal_SNRmin_1064,1 1 1 1,depol_cal_sigMax_1064,2000 2000 2000 2000,rel_std_dplus_1064,0.2,rel_std_dminus_1064,0.2,depol_cal_segmentLen_1064,40,depol_cal_smoothWin_1064,8,isFR,1 1 1 1 1 1 1 1 0 0,isNR,0 0 0 0 0 0 0 0 1 1,is532nm,0 0 0 0 1 1 0 0 1 0,isRR,0 0 0 0 0 0 0 0 0 0,is355nm,1 1 0 0 0 0 0 0 0 0,is1064nm,0 0 0 0 0 0 0 1 0 0,isTot,1 0 0 0 1 0 0 1 1 0,isCross,0 1 0 0 0 1 0 0 0 0,isParallel,0 0 0 0 0 0 0 0 0 0,is387nm,0 0 1 0 0 0 0 0 0 0,is407nm,0 0 0 1 0 0 0 0 0 0,is607nm,0 0 0 0 0 0 1 0 0 1,is1058nm,0 0 0 0 0 0 0 0 0 0 0 0 0,channelTags,,channelTag,[1],FR-total-355 nm,[2],FR-cross-355 nm,[3],FR-387 nm,[4],FR-407 nm,[5],FR-total-532 nm,[6],FR-cross-532 nm,[7],FR-607 nm,[8],FR-total-1064 nm,[9],NR-total-532 nm,[10],NR-607 nm,,minPC_fog,60,TR,0.69 500 1 1 1.09 500 1 1 1 1,overlapCalMode,1,overlapCorMode,3,overlapSmoothBins,8,saturate_thresh,100,heightFullOverlap,500 500 500 500 500 500 500 500 150 150,cloudScreenMode,1,maxSigSlope4FilterCloud,3000000,maxSigSlope4FilterCloud_NR,400000,intNProfiles,120,minIntNProfiles,90,meteorDataSource,gdas1,flagUseLatestGDAS,1,radiosondeType,1,radiosondeSitenum,0,minDecomLogDist355,0.2,minDecomLogDist532,0.2,minDecomLogDist1064,0.2,maxDecomHeight355,8000,maxDecomHeight532,8000,maxDecomHeight1064,8000,maxDecomThickness355,700,maxDecomThickness532,700,maxDecomThickness1064,700,decomSmoothWin355,60,decomSmoothWin532,60,decomSmoothWin1064,120,minRefThickness355,300,minRefThickness532,300,minRefThickness1064,300,minRefDeltaExt355,1,minRefDeltaExt532,1,minRefDeltaExt1064,1,refH_FR_355,,refH_FR_532,,refH_FR_1064,,refH_NR_355,2500 3000,refH_NR_532,2500 3000,minRefSNR355,5,minRefSNR532,5,minRefSNR1064,4,minRefSNR_NR_355,10,minRefSNR_NR_532,5,LR355,50,LR532,50,LR1064,50,LR_NR_355,50,LR_NR_532,50,refBeta355,2e-08,refBeta532,1e-08,refBeta1064,2e-09,smoothWin_klett_355,21,smoothWin_klett_532,21,smoothWin_klett_1064,21,smoothWin_klett_NR_355,13,smoothWin_klett_NR_532,13,smoothWin_raman_355,61,smoothWin_raman_532,61,smoothWin_raman_1064,61,smoothWin_raman_NR_355,25,smoothWin_raman_NR_532,25,maxIterConstrainFernald,20,minLRConstrainFernald,1,maxLRConstrainFernald,150,minDeltaAOD,0.01,minRamanRefSNR355,50,minRamanRefSNR532,20,minRamanRefSNR1064,10,minRamanRefSNR387,20,minRamanRefSNR607,10,minRamanRefSNR_NR_355,50,minRamanRefSNR_NR_532,20,minRamanRefSNR_NR_387,40,minRamanRefSNR_NR_607,20,min_RR_RefSNR1058,0,angstrexp,0.9,angstrexp_NR,0.9,LCMeanWindow,50,LCMeanMinIndx,70,LCMeanMaxIndx,1000,LCCalibrationStatus,[1],none,[2],Klett,[3],Raman,[4],Defaults,[5],History,,quasi_smooth_h,8 8 8 8 8 8 8 8 8 8,quasi_smooth_t,10 10 10 10 10 10 10 10 10 10,IWV_instrument,AERONET,maxIWVTLag,0.0833,tTwilight,0.0347,hWVCaliBase,120,hWVCaliTop,8000,minSNRWVCali,10,clear_thres_par_beta_1064,1e-08,turbid_thres_par_beta_1064,2e-07,turbid_thres_par_beta_532,2e-07,droplet_thres_par_depol,0.05,spheroid_thres_par_depol,0.07,unspheroid_thres_par_depol,0.2,ice_thres_par_depol,0.35,ice_thres_vol_depol,0.3,large_thres_ang,0.75,small_thres_ang,0.5,cloud_thres_par_beta_1064,2e-05,min_atten_par_beta_1064,10,search_cloud_above,300,search_cloud_below,100,overlap532Color,0 255 64,overlap355Color,0 0 255,xLim_Profi_Bsc,-0.05 3,xLim_Profi_NR_Bsc,-0.1 20,xLim_Profi_Ext,-1 150,xLim_Profi_NR_Ext,-1 500,xLim_Profi_WV_RH,0 8,xLim_Profi_WVMR,0 50,xLim_Profi_RCS,0.3 10,xLim_Profi_LR,0 120,yLim_LC_355,0 100000000000000,yLim_LC_532,0 200000000000000,yLim_LC_1064,0 600000000000000,yLim_LC_387,0 100000000000000,yLim_LC_607,0 300000000000000,yLim_LC_355_NR,0 10000000000000,yLim_LC_532_NR,0 20000000000000,yLim_WVConst,0 20,yLim_FR_RCS,0 20000,yLim_NR_RCS,0 3000,yLim_FR_DR,0 15000,yLim_att_beta,0 15000,yLim_att_beta_NR,0 3000,yLim_OC_att_beta,0 15000,yLim_Quasi_Params,0 15000,yLim_WV_RH,0 7000,yLim_Profi_Ext,0 5000,yLim_Profi_LR,0 5000,yLim_Profi_DR,0 20000,yLim_Profi_Bsc,0 20000,yLim_Profi_WV_RH,0 7000,yLim_LC_ratio_355_387,0 2,yLim_LC_ratio_532_607,0 2,yLim_depolConst_355,0 0.2,yLim_depolConst_532,0 0.2,yLim_depolConst_1064,0 50,zLim_att_beta_355,0 15,zLim_att_beta_532,0 5,zLim_att_beta_1064,0 2,zLim_quasi_beta_355,0 8,zLim_quasi_beta_532,0 2,zLim_quasi_beta_1064,0 0.8,zLim_quasi_Par_DR_532,0 0.4,zLim_FR_RCS_355,0.01 15,zLim_FR_RCS_532,0.01 15,zLim_FR_RCS_1064,0.01 15,zLim_NR_RCS_355,0.01 2,zLim_NR_RCS_532,0.01 2,zLim_VolDepol_355,0 0.2,zLim_VolDepol_532,0 0.3,zLim_VolDepol_1064,0 0.3,colormap_basic,labview,PI,,PI_affiliation,,PI_affiliation_acronym,,PI_address,,PI_phone,,PI_email,,Data_Originator,,Data_Originator_affiliation,,Data_Originator_affiliation_acronym,,Data_Originator_address,,Data_Originator_phone,,Data_Originator_email,,comment,,logbookFile,/data/level1a/polly/logbook_polly_Arielle_til_2018.csv,radiosondeFolder,,calibrationDB,arielle_calibration.db,imgFormat,png,partnerLabel,,prodSaveList,[1],overlap,[2],aerProfFR,[3],aerProfNR,[4],aerProfOC,[5],aerAttBetaFR,[6],aerAttBetaOC,[7],aerAttBetaNR,[8],WVMR_RH,[9],volDepol,[10],quasiV1,[11],quasiV2,[12],TC,[13],TCV2,[14],cloudinfo,,
@cristoferjimenez @ZPYin Thanks a lot! 🥇 I will implement it the next days. Then, together with the PPC commit, we have a full reproducible processing chain!
Looks good. One question concerning the deadtime-correction polynomial matrix (keyword: "dt").... at the moment the matrix is written channel-wise and polynomial-wise all together. This is very hard to read and distinguish in the end. Is it possible to output it with brackets, like: [polynom 1.channel][polynom 2.channel][polynom 3.channel] ....?
Hi all,
i implemented Zhenpings solution to the PPC, see https://github.com/PollyNET/Pollynet_Processing_Chain/tree/Polly_Config_Storing or the repsective pull request: Polly_Config_Storing
@ulysses78 @ZPYin @cristoferjimenez if you have a solution for @ulysses78 idea, please feel free to change the code and update it to the branch before we merge it with dev.
as it is working since a while and nobody had a better idea I close it for now