Automatic Download of Scan Frequncies from SondeHub?

[ZigiWalter]

Hi,

First - thanks for everyone contributing to this great project!

In the area I live in, lunch sites use varying frequencies, so I’m looking for a way rdz_ttgo_sonde would automatically scan the frequency of the current lunch during a chase, but without having to program it over WiFi. Such “automatic discovery” mode is not currently supported, right? Or maybe I've missed it?

One solution I thought of is for the FW to download a list of the in-flight sondes around the user from SondeHub, using the below API call (which already filters the sonde list by distance and time): https://github.com/projecthorus/sondehub-infra/wiki/API-(Beta)#sondes I did a quick-and-dirty PoC (sorry – I’m not a programmer…), and it actually works. Obviously, I didn’t cover all corner cases nor considered resource limitation that might pop up.

But regardless of any specific implementation – is that a desired direction that can eventually go into the main trunk of rdz_ttgo_sonde? Or maybe you think we shouldn't go that way? (due to limited memory, having a better solution, or some other reason?)

Thanks.

[LukePrior]

This sounds like an interesting concept do you have the code for your proof of concept?

While I think it would be cool it is my personal belief that attention towards an automatic frequency detection system such as present in auto_rx would be more beneficial, I'm not even sure if that is possible on this hardware @dl9rdz ?

[ZigiWalter]

Sure - I'll add some comments to the code (such that not only I would be able to read it...) and share.

[ZigiWalter]

@LukePrior - here is the entry point: https://github.com/ZigiWalter/rdz_ttgo_sonde/blob/devel/RX_FSK/RX_FSK.ino#L3019 (Edit: I've just realized that a link to a specific line number is rendered useless very fast... instead - search for: getSondehub(shclient,1000) in RX_FSK.ino)

All relevant code is in my_sondehub.cpp (yep, sorry for that... I didn't want to make changes all over the place before I understand whether this approach makes sense)

Let me know if you need any clarifications.

[dl9rdz]

Ok, looks interesting and likely feasible with little program memory footprint.

I will be on holiday in one week from now and probably mostly busy with other task the days before going on holiday, so if you don't see any updates from my side the next 3-4 weeks, its not a lack of interest. But I want to have a closer look on your code and maybe make a few minor changes. Using as little dynamic RAM allocation as possible has shown to be quite useful for stability on the ESP32.

[LukePrior]

I see that this is merged and I just gave it a try here are my thoughts:

The configuration entry is very unclear especially for a newcomer, I would recommend that a new section is created titled "Automatic Sonde Frequency Detection" or something along those lines. The next thing would be a note saying that a GPS position is required either by setting it or using a connected GPS/phone app.

The three settings should each be broken into their own separate entry with a small explanation of what they do and a new entry added to disable/enable:

• Enabled (0 or 1)
• Interval (default 60 so every hour, set a minimum limit in code of 5 minutes to avoid overwhelming Sondehub servers)
• max distance (this is fine at 100km keep as is)
• maxage (I would change this to be in hours and set the default at maybe 24 hardcode a limit of 7 days/168 hours)

I think the user should be able to run this at any time from the web interface on the QRG page as currently, it is all too hidden with no indication it is running.

When I got it all setup on a new installation I noticed that it did not replace the default configuration file values and instead appended to the bottom. I think any default values which are not enabled should be replaced see the image. I didn't have enough room to get more than 1.

There needs to be some sort of rounding as 402.5 and 402.501 should not be two separate entries. I think rounded everything to 1dp should do the trick.

[ZigiWalter]

Thanks for the time going into this! Indeed - the behavior I chose for the PoC is populate vacant entries at the end of the list, to minimize "interference" with original behavior. When I run it, it definitely ends up using more than a single entry. I think the code in the main branch changed quite a lot since (also causing some conflicts). Let me have a look.

Thanks again.

[ZigiWalter]

I had to fix some compilation issues, due to changes in the main branch. Anyway - I do still see more than a single sonde automatically downloaded (screenshot below). Please let me know if there is any way I can help.

[dl9rdz]

Yes, I did quite a number of things differently in the implementation, mainly because I wanted it to be able to run in the background so the decoder runs normally while retrieving the reply, and also to minimize code size. It is also synced with data uploads (i.e., the request is always handled in between two data uploads if SondeHub upload is active, never in parallel.)

@LukePrior The config can certainly be broken into multiple fields. For now I wanted to keep it short, as the config page already is pretty large, close to causing out of memory issues, as currently the full page is constructed on the TTGO dynamically. I will offload some config functionality to Javascript that can simply put into a file, so we can save considerable RAM here, and in that process change that into three (or four) fields. Probably some other things can be re-arranged as well. The SondeHub config should be next to the other data feed configurations, not separated by the hardware config.

Not sure a dedicated "active" checkbox is really needed, or if using 0 for the interval would be good enough to disable, along with a comment. Enforcing a minimum of 5 minutes sounds like a good idea, sure.

As for using slots marked as inactive, my rational is that if the user has explicitly created an entry with some label, he did so intentionally, and such entries should not be overwritten. What my current code does is that if there is a deactivated entry on a frequency reported by SondeHub, this entry is activated instead of creating a new entry. It would make sense though to have more than a single empty slot in the default configuration.

Regarding rounding, what I currently do is consider two frequencies identical if they differ by less then 1.5 kHz. Maybe this can be increased slightly. But if by 1dp you mean rounding to full 100 kHz, that would be too coarse-grained. Most RS41 I see operate on full 100 kHz step frequencies, but other types frequently do not, and they will not decode with a setting 10 kHz off.

[LukePrior]

I will offload some config functionality to Javascript that can simply put into a file, so we can save considerable RAM here, and in that process change that into three (or four) fields.

That sounds good.

Not sure a dedicated "active" checkbox is really needed, or if using 0 for the interval would be good enough to disable, along with a comment. Enforcing a minimum of 5 minutes sounds like a good idea, sure.

I like the idea of having a dedicated box as it allows us to set a default interval, it is quite an abstract concept so people might not know what to set the time to.

It would make sense though to have more than a single empty slot in the default configuration.

Yeah this would solve it, if you made at least 5 spots free I think that would work.

what I currently do is consider two frequencies identical if they differ by less then 1.5 kHz.

Oh I didn't realise in that case please ignore my comment.

What do you think about adding a button to the web-page to force start the process would this be possible?

[ZigiWalter]

Do you think the minimal distance of downloaded sondes should be configurable too? Maybe instead of having a check-mark to enable/disable this feature, there would be an integer number, where "0" means disable, and other values specify the min. distance in Km? Also - how would you prevent downloading a sonde again in the next interval? Is it by examining all existing entries for frequency and type?

Thanks.

[LukePrior]

Do you think the minimal distance of downloaded sondes should be configurable too?

I don't think this is necessary

I am a large fan of making things simple so as to not intimidate new users as such I am a fan of making enable a simple 0/1.

Also - how would you prevent downloading a sonde again in the next interval?

What do you mean by this it is fine if we download the same sondes again as duplicates are checked for.

[ZigiWalter]

I don't think this is necessary

I am a large fan of making things simple so as to not intimidate new users as such I am a fan of making enable a simple 0/1.

Makes sense. So can we go with 200Km as a default? or is it too large in "dense" sonde environments like Europe...?

What do you mean by this it is fine if we download the same sondes again as duplicates are checked for.

Oh, I didn't spot that. Will look again.

Thanks!

[LukePrior]

So can we go with 200Km as a default?

I think this is too large but simply from a perspective of what the antenna will get. When you have a good setup it might work out that far fine but I think 100km is a good default and advanced users who know that they have a good antenna can extend this. I think it would also be good to set a limit of maybe 700km to avoid getting too many results

[ZigiWalter]

I have easily exceeded the 100Km limit so I thought it might be too low, but maybe it's because of my home-made antenna and not a typical use-case :-)

[LukePrior]

I just updated to the latest version and love the new settings page. The whole processes is significantly clearer.

I'm not sure how hard it would be to add with the new settings page design but it would be handy to have a little bit of text after the SondeHub frequency import describing what it does.

I also think the maximum time should be increased to 48 hours as for those outside Europe we might not have as many launches.

I also saw that you moved SondeHub GPS to a common location at the top of the file. I think another little blurb under the SondeHub settings section saying that these fields should be set would be useful.

I think the rounding should be increased from 1.5Hz to maybe 3Hz as I was getting entries for 402.500 and 402.502.

Some unrelated stuff but I think the SondeHub host field should be removed as noone should actually every need to change it.

[dl9rdz]

Basically, the new settings page is now all in cfg.js, i.e. simple JavaScript, outside of the ESP32 code, and can more easily be modified (even without having to update any code). Any non-excessive extensions would be technically feasible, like adding a table row below the headings with some explanations. All rows below a heading have to have "cfgpanel" class in the tr elements, for the expand/collapse function to work properly.

However, additional explanation should not be too long in order to not negatively affect the usability on small smartphone displays.

Another alternative (or additional option) might be adding an info button to the config headings with links to the corresponding Wiki pages on github with detailed explanations.

[LukePrior]

Another alternative (or additional option) might be adding an info button to the config headings with links to the corresponding Wiki pages on GitHub with detailed explanations.

I think this combined with a short 1 sentence explanation for relevant sections would do the trick nicely.

[dl9rdz]

So can we go with 200Km as a default?

I think this is too large but simply from a perspective of what the antenna will get. Just accidentally saw that my T-Beam easily picks up S2011305 270km away. Just a X30 (70cm ham) antenna on balcony, long cable, no filter nor preamp. But anyway, default value is at 150km now.

[F5MVO]

Very nice function, but 700kms is better for coverage the South France here, who can do more can do less. I use directive antenna with LNA

Good job

[F5MVO]

Hi, i have a new entry, 401.005 for 401.000

[ZigiWalter]

I've somehow missed the updates, and only now have I realized that this feature made it in... @dl9rdz , @LukePrior , and whoever else was involved - thanks! Your efforts are greatly appreciated.

[dl9rdz]

I am seeing one issue, which are stations reporting wrong frequencies to SondeHub. Probably, these are receiver stations with low-quality and uncalibrated RTL-SDR. For example ok5tvr is reporting most RS41 I can receive (such as T1330867) with a frequency about 13 kHz off. This is strange in any case, because (based on aprs.fi raw data) the station is using autorx, and autorx should be reporting the frequency from decoded telemetry if available, and SDR frequency only if that is not available.

I guess there is not much we can do about on the TTGO that. Maybe we could add a check if we have received some ID on some frequency before, and in that case ignore SondeHub reports for the same ID on different frequencies. I don't see any other good option. Possibly, this should be handled on the server side, if multiple stations are reporting some sonde, frequency outliers could be filtered out.

I can make the minimum frequency offset within which additional entries are created, currently at 3 kHz, configurable or larger.

Regarding the other comment, I can increase the upper limit of the range, 700km instead of 500km is fine. I don't think anything larger than that makes much sense practically.

[73-de-LZ]

At Hansi DL9RDZ and all involved. First I want to say BIG THANKS from all my heart, it is just GREAT! Thousands UFB! *Ultra Fine Business! It is very well documented and it is a good example how an project should be developed and documented. The opportunity to compile the source with Arduino IDE or just flash the bin is also very important. I saw it month ago and while i play with it I can only be happy and happy again about any detail refined to perfection! QRG import is fine in devel20220123, filtering minimum frequency offset is OK at 3 kHz, at least we can see time to time that the function is working, if it is configurable would be great for those who hate the spam.

[gustavc39]

Hi, I'd like to begin by saying a big THANKS to the developers and maintainers of this software which is very well designed and works great. As a new user I found it difficult to make this feature work because there's no explicit indication that you need to reserve one entry at the bottom of the QRG table set with frequency 0.000 and empty comment for this to work properly, other than this thread. I think that it would be very helpful to include this point in the wiki. Thanks again and keep it up the good work!