SuperDARN
Make FoV routine
A preliminary version of the routine that automatically determines the backscatter return direction. This partially addresses #135.
Test results using a fitacf 3.0 file:
make_fov -cn a ~/Programs/Data/SuperDARN/Downloads/19960916.0400.00.han.fitacf3 > han.tst_fov
This creates an ascii file with 13727 lines and the following columns:
#STID DATE TIME BMNUM BMAZM CPID INTT_SC INTT_US NAVE FRANG RSEP RXRISE FREQ NOISE ATTEN CHANNEL NRANG RG GFLG FOVFLG FOV_PAST GRPFLG GRPNUM GRPID P_0 P_0_ERR V V_ERR W_L W_L_ERR P_L P_L_ERR PHI0 PHI0_ERR ELV ELV_LOW ELV_HIGH VH VH_ERR VH_METHOD REGION HOP DIST MED_P_0 MED_P_0_ERR MED_V MED_V_ERR MED_W_L MED_W_L_ERR MED_P_L MED_P_L_ERR MED_PHI0 MED_PHI0_ERR OPP_ELV OPP_ELV_LOW OPP_ELV_HIGH OPP_VH OPP_VH_ERR OPP_VH_METHOD OPP_REGION OPP_HOP OPP_DIST
Re-creating one of the figures from my paper leads to similar results. This scan is from a different time that was created during various spot checks.
Before moving this from draft status we need to:
- [x] merge #444
- [x] Move general utility functions into appropriate libraries
- [x] Create a fitacf-esq non-ascii output file option
- [x] Finalize command-line options
I am having issues compiling this on CentOS release 6.10.
Less worrying: I am getting this warning that may or may not need to be addressed:
$RSTPATH/include/base/rtypes.h:49: warning: ISO C90 does not support ‘long long’
$RSTPATH/include/base/rtypes.h:50: warning: ISO C90 does not support ‘long long’
The error (shortened for legibility):
$RSTPATH/lib/libfitacf.3.0.a(llist.o): In function `llist_sort':
llist.c:(.text+0x2d2): undefined reference to `pthread_rwlock_unlock'
llist.c:(.text+0x2e6): undefined reference to `pthread_rwlock_wrlock'
$RSTPATH/lib/libfitacf.3.0.a(llist.o): In function `llist_reverse':
llist.c:(.text+0x355): undefined reference to `pthread_rwlock_wrlock'
llist.c:(.text+0x365): undefined reference to `pthread_rwlock_unlock'
$RSTPATH/lib/libfitacf.3.0.a(llist.o): In function `llist_concat':
... et cetera ...
$RSTPATH/lib/libfitacf.3.0.a(llist.o): In function `llist_create':
llist.c:(.text+0xe26): undefined reference to `pthread_rwlockattr_setpshared'
llist.c:(.text+0xe36): undefined reference to `pthread_rwlock_init'
llist.c:(.text+0xe42): undefined reference to `pthread_rwlockattr_destroy'
collect2: ld returned 1 exit status
make: *** [make_fov] Error 1
I tried added the pthread.h and llist.h libraries to the routines that might possibly need them, but it didn't change anything.
Ok @ecbland and @egthomas I am a bit stuck on the last step: creating a binary file. I can write a binary without issue, but I can't read it back in. This could be:
- a problem writing a valid file, or
- just a reading problem.
I added a utility fitmultbsidtoascii to help test this. Would appreciate help debugging, as I am stuck and lack dmap knowledge/experience.
@aburrell if it is a valid dmap-format file then you should be able to use
dmapdump [filename] > [output_filename]
to see all of the scalar fields in the file in plain-text format, or
dmapdump -d [filename] > [output_filename]
to see both the scalars and contents of any arrays.
Ok @egthomas the good news is that the output file is valid! The bad news is I still have things I need to fix 🙀 But that's a better problem to have.
@aburrell, I am a bit confused about how to test this routine. I have installed the branch and ran make_fov on a FITACF file. The output seems to be in DMAP format rather than ASCII mentioned at the beginning of the issue. I can also see that it is a FITACF-like format but with extra fields. Could you please provide a bit more comprehensive instructions for a non-initiated reviewer like me? :-)
@pasha-ponomarenko before I give you detailed instructions (which I am happy to do) could you try following the examples I wrote up in the local routine docs? I think it would be good to have someone unfamiliar with the routine, but a competent RST user try to follow them so I can use your feedback to make them clearer.
But to answer a quick question you raised, there are two output formats: ascii and a dmap binary. It is basically fitacf + some fields. Not all fields are used by this routine, some are there in preparation for the rest of the backscatter ID processing that I will code up following this pull.
@aburrell sorry what are the new fields called?
Make sure to add them to pyDARNio so people can read them :)
@aburrell, do you mean this one? https://github.com/SuperDARN/rst/blob/make_fov/codebase/superdarn/src.bin/tk/tool/make_fov.1.0/docs/make_fov.doc.xml
When I try to open it in a browser, I get this error:
It might be something very simple, but ...
I do mean that one! Unfortunately I do not know XML, so I made the docs by copy-pasta + add sauce. Can someone with XML knowledge take a look at that file and let me know what might be wrong? @pasha-ponomarenko do you know anyone with this skill set?
@mts299 the new fields are all in the new structure housed in $RSTPATH/codebase/superdarn/src.lib/tk/fitmultbsid.1.0/. All the parameters are explained in the docs in that file, so if you don't mind trying to read them and coming back with questions, I'd appreciate your partial review!
I will do it next week (maybe), currently cannot stare at a screen due to a concussion. Was just looking for quick things to do or ask.
Apparently, there is a know issue with Github: one should never download a web file from Git as it will contain the Git webpage information as well: https://github.com/pm4py/pm4py-core/issues/247#issuecomment-907742973
However, when I tried to open the same file from an RST package installed on my Linux desktop, it just showed a "raw" ASCII content and the following error(?) message: "This XML file does not appear to have any style information associated with it. The document tree is shown below. " This happens on both Linux and Windows.
@aburrell, are you able to see the document properly in your web browser?
are you able to see the document properly in your web browser
No, but I don't know how to open up any of the RST xml docs. So it's a matter of me not knowing how to create something that I could open. Do you just try to open the xml files in a web browser?
So, opening any of the files in a web browser doesn't do anything but display the code. I guess the good news is that my xml behaves the same as the rest of the xml. None of them open.
Given that it seems none of us have the knowledge to resolve this issue, here's what you need to run the code:
Create ascii output: make_fov -ascii
Because you need to look at the range and azimuthal variations, you need to separate frequency bands for the analysis. For best results, use the TDIFF suited to the choses radar channel and frequency!
To convert the dmap binary to ascii, use fitmultbsidtoascii
Yep, the XML business looks like a separate issue. :-( Thanks for the clarification. Do you have a description of the extra fields in a more robust format like DOC or PDF? Or simple ASCII?
To view the documentation in a web browser, I think you first need to run make.doc from the root directory of RST.
cd $RSTPATH
make.doc
cd doc/html
google-chrome index.html & # or another web browser of your choice
You can then navigate through the html documentation in the web browser.
The documentation for the make_fov binary is here: $RSTPATH/doc/html/superdarn/src.bin/tk/tool/make_fov/index.html
No, I put all the documentation in xml since that was the RST standard.
The most important thing is FOVFLG, which is 1 for the front FoV, -1 for the rear FoV, and 0 if a FoV could not be cleanly identified. GFLG is also updated so that it is -1 if it was formerly groundscatter and doesn't look like groundscatter.
@aburrell Now I am confused. Here's a screenshot from following the instructions I gave in my previous comment. Is this what @pasha-ponomarenko was looking for?
Is this what @pasha-ponomarenko was looking for?
Yes, @ecbland that is it!
@ecbland, thanks a lot, I can see this page after I run make.doc ;-) I was able to process a FIATCF file last night, and I wanted to figure out what is *.fov data format so that I would be able to visualise it one way or another. From dmapdump output I saw that it is similar to FTITACF but with extra fields.
@aburrell, I might miss something, but there seem to be fewer fields in a binary output of make_fov than in its ASCII one, e.g., MED_* and GRPFLG, GRPNUM, GRPID are all missing.
Yes, those fields are for future routines down the backscatter ID stream. Because the binaries are the default output for RST, I wanted them to be streamlined and only have the appropriate information. Since the ASCII output is primarily diagnostic, I added less logic steps to remove currently unused parameters.
From: pasha-ponomarenko @.> Reply-To: SuperDARN/rst @.> Date: Wednesday, February 9, 2022 at 17:53 To: SuperDARN/rst @.> Cc: "Burrell, Angeline" @.>, Mention @.***> Subject: Re: [SuperDARN/rst] Make FoV routine (#449)
@aburrellhttps://github.com/aburrell, I might miss something, but there seem to be fewer fields in a binary output of make_fov than in its ASCII one, e.g., MED_* and GRPFLG, GRPNUM, GRPID are all missing.
— Reply to this email directly, view it on GitHubhttps://github.com/SuperDARN/rst/pull/449#issuecomment-1034281370, or unsubscribehttps://github.com/notifications/unsubscribe-auth/ABVYF7QPA6ZDYTJI7C4M3DDU2LV6DANCNFSM5BWQM2NQ. You are receiving this because you were mentioned.Message ID: @.***>
OK, here are some preliminaries. I used 20150223.2001.00.sas.fitacf (v3, twofrequency mode) which contains both ionospheric and ground scatter. I also selected SAS data because the ground scatter here is normally dominated by signals received through the grating lobe. Here are plots for the 10 MHz band of the twofrequency mode. Importantly, for this interval there was no detectable tdiff offset for this frequency band. The data show both ionospheric and ground scatter, and ground scatter coming from rather high elevation angles which is indicative of the "back" lobe scatter.
I processed this data for frequencies between 10 and 11 MHZ using following command line options:
make_fov -vb -tfmin 10000 -tfmax 11000 20150223.2001.00.sas.fitacf > 20150223.2001.00.10.sas.fov
Then I converted DMAP *.fov file into NetCDF with following steps:
- Analysed DMAP structure and produced supplementary files:
dmaptocdl 20150223.2001.00.10.sas.fov fov.cdl fov.cdlmap - Adjusted the field lengths in the "mask" file fov.cdl to 75.
- Generated an empty NetCDF file
ncgen fov.cdl -o fov_10.netcdf - Filled the empty file with data from DMAP file:
dmaptoncdf 20150223.2001.00.10.sas.fov fov.cdlmap fov_10.netcdf
After that used IDL to read and plot NetCDF FOV data:
Velocity
Elevation
FOV
GSCT
SCT
My understanding is that the majority of the 1.5 ionospheric scatter above rage gate 40 is removed due to one of the selection criteria (exceeding the maximum virtual height?). The FOV determination seems to be a bit shaky for the majority of the ground scatter before 21:30 UT. I guess it might be related to an apparent partial overlap between the front- and "back"-lobe ground scatter. Indeed, the closer edge of the GS band between gates 18 and 25 shows very high elevation angle values in excess of 35 deg followed by a sharp boundary with elevation angle values dropping to 20-25 deg at gates ~25-32.
@aburrell, please correct me if I am wrong here. BTW, I am not sure what SCT flag means.
Ok, I played around with things and the main reason some things aren't working well is the propagation grouping. I applied the HAN defaults (basically the same, but -fhmax 900) and got much more of the ionospheric scatter back. Fine tuning the propagation virtual height ranges would further improve the results. I added a note to the docs to make sure users know how important this is, and know where the defaults come from.
Looking at the azimuthal variations, it makes sense that much of the groundscatter returns from the front FoV, since the variations are consistent across the beams.
BTW, I am not sure what SCT flag means.
It's the sct parameter from the fitacf structure, I believe it just flags whether the scatter is valid or not.
It's the sct parameter from the fitacf structure, I believe it just flags whether the scatter is valid or not.
I don think there is such a parameter in the fitacf structure https://radar-software-toolkit-rst.readthedocs.io/en/latest/references/general/fitacf/, that is why I asked :-P It does not look like a quality flag...
@aburrell , What language did you make your plots in? Python? IDL? Is there a way to integrate some plotting into this package (or utilise some existing RST plotting routines) in order to make tester's/user's life a bit easier? :-{
I don think there is such a parameter in the fitacf structure
There is! Just grep "sct" in $RSTPATH/include/superdarn/*.h and you'll see which structures it's in. You can also examine the code to see how it's used. It may not be included in the user-description of the output, but it's a code structure parameter.
What language did you make your plots in
Python. I don't know how to plot in C, so I won't be adding this to RST. They're just RTI plots like the ones you made, but further separated by hop or propagation path. The plots you made were fine, they confirmed the code is working as it should be!