trunk-recorder
trunk-recorder copied to clipboard
robotastic
Auto Adjust PPM
It would be nice to leverage the LTE-Cell-Scanner module by rxseger to automatically adjust the PPM nightly for full duty cycle trunk-recorders or provide a "run once" option to config all PPM's VS doing it manually via a GUI.
An ugly solution for the group is below. Provided free by myself to the community.
Below is an ugly perl script that will leverage the rxseger LTE module to load your trunk-recorder config file. Query LTE towers in your area and estimate your PPM for each of your RTL cards.
This could be ran nightly, weekly or just one time to get an estimated PPM for all your cards at once VS having to manually configure each card.
This script is provided FREE with no WARRANTY and it's VERY UGLY - But it works.
NOTES: Only tested on Ubuntu - No windows version. Only tested on RTL cards - You can tweak as needed. YOU MUST STOP your trunk-recorder (free up your RTL cards) for this script to run.
Instructions for linux users:
-
Install https://github.com/rxseger/LTE-Cell-Scanner
-
Make sure it works by running something like this (XX = your RTL card index number): CellSearch --freq-start 739e6 --freq-end 739e6 -iXX
-
Once you find a freq range that works for you - Cut and past your freq values and keep them to the side.
-
cut and past the below code.
-
vi your_script_name.pl AND past this AS-IS code.
-
Modify the TOP of the program to fit your needs (specify how many RTL cards you have - and your frequency range (what you put to the side in step 3). You can keep verbose/debug on if you want. Up to you.
-
Make executable (or whatever perms u want) - chmod 755 your_script_name.pl
-
./your_script_name.pl [your_json_config_file.json] (no brackets)
-
Review output your_json_config_file.json.new
-
I would run this outside your main directory to be safe. Tweak and share this ugly script for others to use as you wish.
Each LTE tower can take 6-30 seconds to query. This is repeated for EACH card you have. So for 7 RTL cards - It could take up to 2 minutes.
Depends on: rxseger's LTE-Cell-Search rtl_sdr (apt-get install rtl-sdr)
UGLY SCRIPT
#!/usr/bin/perl
### This script provided free
###
### This script is UGLY and has no warranty or claims of functionality. USE AT YOUR OWN RISK! I CAN'T BE RESPONSIBLE FOR ANYTHING!
###
###
### Purpose: Loads trunk-recorder like json config file.
### Uses the ppm values found from LTE-Cell-Scanner and writes them to config file.
###
### If multiple ppm values are returned from cell query - They are added and devided into an average and output to config file.
###
### This script can be used to create an automated process to update config file.
###
### NOTE: YOU MUST SHUTDOWN YOUR TRUNK-RECORDER to free up the RTL cards for this to work!!! Each radio card you have will take about 6-30 seconds
### to query and return results. For 7 cards you are looking at almost 2 minutes
###
### Be sure to adjust the freqencyRange to meet your part of the country.
###
### Your new file is output with .new extension. You can then copy it where you need it or modify this routine to fit your needs.
###
### Syntax:
### program_name.pl [config_file.json]
##################
### Auto PPM setting
###
### This program is ONLY designed to work with git clone https://github.com/rxseger/LTE-Cell-Scanner
###
### See document: https://www.surviveuk.com/wp-content/uploads/2016/07/The-Hobbyists-Guide-To-RTL-SDR-Carl-Laufer.pdf
###
##############################
### PROGRAM SETTINGS
##############################
my $howManyRtlCardsYouHave=6;### Indexing starts at 0. So if you have 7 cards, you enter 6.
my $deviceToErrorSpacing = 10; ### This is the estimated spacing between your device tag and error tag inside your json file. This number should be equal or greater than your spacing. 10 should be good. ****NOTE: Your "error" tag MUST exist ABOVE the "device" tag within your config file!
my $whatIsYourRtlName="RTL";### This is part of what your card is named. See rtl_test to see the list of names. Example: 0: Realtek, RTL2838UHIDIR, SN: 004
my $freqencyRange="--freq-start 739e6 --freq-end 739e6";### Enter the range of LTE frequencyes for your part of the country.
my $debug=1;### Set to =1 for debug output
my $verbose=1;### Set to =1 for verbose output;
##################
### DO NOT EDIT BELOW HERE
##################
use strict;
use warnings;
use Data::Dumper;
$Data::Dumper::Indent = 1;
my $cmd;
### Load config file
my $filename = $ARGV[0] || "";
### Open config file;
print "Loading file [$filename]\n" if ($verbose || $debug);
open( my $fh, '<', $filename ) || die "Can't open $filename: $! - Syntax is: program_name.pl [config_filename]\n\n";
my @inFile = <$fh>;
close $fh;
###Make sure infile has a json related device tag that we need. If not, lets throw an error and halt.
my $foundIt;
foreach my $line (@inFile){
if ($line =~ m/\"device\"/i){
$foundIt=1;
}
}
if (!$foundIt){
print "filename [$filename] doesn't appear to have a device json tag. Halting.\n\n";
exit;
}
print "File [$filename] passed initial testing.\n\n" if ($verbose || $debug);
### We load this command and replace X with our value
my @cellResults;
foreach (my $i=0; $i<=$howManyRtlCardsYouHave; $i++){
$cmd="CellSearch $freqencyRange -iXX 2>&1";## > results_739_740_cardXX.txt";
$cmd=~s/XX/$i/;
print "Calling system command [$cmd]\n\n" if ($verbose || $debug);
my @cmdResults = qx~$cmd~;
print "Finished with system command.\n\n" if ($verbose);
print "Dumper of command results:" . Dumper(@cmdResults) . "\n\n" if ($debug);
my @innerResults;
### Loop cmdResults for what we want.
foreach my $line (@cmdResults){
chomp $line;
if ($line =~ /^\s*FDD/){
### Get our ppm
my ($ppm) = $line =~ m/.*\s(.*)$/;
push(@innerResults, $ppm);
}
}
print "PPM Results for card index [$i]:" . Dumper(@innerResults) . "\n\n" if ($debug);
push(@cellResults, \@innerResults);
}
### GET rtl device index.
$cmd="rtl_sdr -d0 2>&1";
print "Calling system command [$cmd]\n\n" if ($verbose || $debug);
my @cmdResults=qx~$cmd~;
print "Finished with system command.\n\n" if ($verbose);
print "Dumper of command results:" . Dumper(@cmdResults) . "\n\n" if ($debug);
### Loop cmdResults
my @outputResults;
foreach my $line (@cmdResults){
chomp $line;
### Pull out index and SN
if ($line =~ m/SN\:\s+\d+/){
my ($indexNum, $serialNum) = $line =~ m/.*(\d)\:.*$whatIsYourRtlName.*SN:\s+(\d+)$/;
if ($indexNum <= $howManyRtlCardsYouHave){
my $ppmResults;
foreach my $innerResults ($cellResults[$indexNum]){
foreach my $ppm (@$innerResults){
$ppmResults.=$ppm."|";
print "INDEX[$indexNum] - PPM [$ppm]\n\n" if($debug);
}
}
### Reove trailing |;
chop $ppmResults;
push(@outputResults, "$indexNum,$serialNum,$ppmResults");
print "INDEX [$indexNum] - SN: $serialNum = ppm of [" . $ppmResults . "]\n\n" if($debug);
}
}
}### END foreach $line
###Output our results
if($debug){
foreach my $item (@outputResults){
print "$item\n";
}
}
modifyConfigFile($filename, \@outputResults, \@inFile);
sub modifyConfigFile{
my ($filename, $inArray, $configFile)=@_;
my @ppmDataArray = @$inArray;
my @configFile = @$configFile;
### Holder for our output
my @newConfigFile;
### Loop our existing config file
for(my $i=0; $i<=$#configFile; $i++){
chomp($configFile[$i]);
my $currentLine = $configFile[$i];
##chomp($currentLine);
my $deviceValueType;
##print "FOOLINE - [$i] - $currentLine\n";
### See if our current line is our device flag of config file.
if ($currentLine =~ m/\"device\"/i){
### See if we have an index value OR a rtl= value.
my $deviceValue;
my $deviceValueType;
if ($currentLine =~ m/rtl\=/i){
($deviceValue) = $currentLine =~ m/.*\"rtl=(.*)\,.*\"$/i;
$deviceValueType = "serial";
}
else{
($deviceValue) = $currentLine =~ m/.*\:.*\"(.*)\"$/i;
$deviceValueType = "index";
}
### See if our ppmDataArray contains any index for our current deviceValueType
foreach my $ppmItem (@ppmDataArray){
print "ppmItem: $ppmItem \n\n" if($debug);
my $foundMatch;
my ($indexNum, $serialNum, $ppmValues) = split(/\,/, $ppmItem);##($ppmItem->[0], $ppmItem->[1], $ppmItem->[2]);##split(/\,/, @$ppmItem);
print "deviceValueType [$deviceValueType] deviceValue [$deviceValue] - serialNum [$serialNum]\n" if($debug);
if ($deviceValueType eq "index" && $indexNum eq $deviceValue){
print "FOUND INDEX MATCH - Our ppmValues are [$ppmValues]\n\n" if($debug || $verbose);
$foundMatch=1;
}
elsif ($deviceValueType eq "serial" && $serialNum eq $deviceValue){
print "FOUND SERIAL MATCH - Our ppmValues are [$ppmValues]\n\n" if($debug || $verbose);
$foundMatch=1;
}
### Modify our error item inside our json file by walking backwards.
if ($foundMatch){
my $foundIndexLineNumber;
foreach (my $x=$i; $x>($i-$deviceToErrorSpacing); $x--){
my $lookbackLine = $configFile[$x];
chomp($lookbackLine);
print "[" . ($x) . "] - Lookback line: $lookbackLine\n\n" if($debug);
last if ($lookbackLine =~ m/\"center\"\s*\:/);### We made it to top of list without finding our error line.
if ($lookbackLine =~ m/\"error\"\s*\:/i){
my $howFarBack = ($i - $x);
$foundIndexLineNumber = ($i - $howFarBack);##$x)+1;
print "lookbackLine = [$lookbackLine]\n\n" if($debug);
##sleep 2;
print "i[$i], x[$x] - foundIndexLineNumber = [$foundIndexLineNumber]\n\n" if($debug);
### Chage original line with new line.
my $averagePpmValue=averagePpmValues($ppmValues);### Averages out ppm values.
my $oldLine = $configFile[$foundIndexLineNumber];
my $newLine = "\t\"error\" : $averagePpmValue\,";
$configFile[$foundIndexLineNumber] = $newLine;
print "Line [$foundIndexLineNumber] - oldLine [$oldLine] \n\n" if($debug);
print "Line [$foundIndexLineNumber] - newLine [$newLine]\n\n" if($debug);
last;
}
}
last;
}### END foundMatch
}### END foreach my $ppmItem
}### END - if ($currentLine =~ m/\"device\"/i){
}### - for(my $i=0; $i<=$#configFile;
my $newFileName = $filename . ".new";
print "Creating new file [$newFileName]\n" if ($verbose||$debug);
### Save our modified file
open( my $fh, '>', $filename . ".new" ) or die "Can't open $filename: $!";
foreach my $line (@configFile){
print $fh $line . "\n";
print $line . "\n" if ($debug);
}
close $fh;
print "Finished writing new file.\n\n" if ($debug || $verbose);
}### END modifyConfigFile
sub averagePpmValues{
my $in=$_[0];
print "AVERAGE [$in]\n\n" if($debug);
### Return value if no delimiters found.
if ($in !~ m/\|/){
print "Single item found. REturning [$in]\n\n";
return $in;
}
### Split our items.
my @items = split(/\|/, $in);
my $howManyItems = $#items+1;### Arrays start at zero.
### Multiple our values.
my $sum;
foreach my $item (@items){
print "PPM ITEM [$item]\n\n" if ($debug);
$sum += $item;
}
print "PPM Total before averaging [$sum]\n\n" if ($debug || $verbose);
### Devide by count.
my $ppm = $sum / $howManyItems;
print "Devide by $howManyItems - Resulting PPM: $ppm\n\n" if($verbose || $debug);
return $ppm;
}### END - averagePpmValues
print "DONE..\n\n\n" if ($verbose);
1;
I agree, that would be nice.
On the idea of automatically setting PPM, I found this fork https://github.com/kkotowick/trunk-recorder which appears to have implemented such a feature: https://github.com/kkotowick/trunk-recorder/commit/3584e844815b052546e6985baf106380e7ad7cb2
After cherry-picking that commit onto my own fork https://github.com/CrimeIsDown/trunk-recorder/tree/crimeisdown-custom I added the following to my config, in an object in the sources array:
"ppm_adjust_interval": 1,
Now the PPM will be tuned from -20 PPM to 20 PPM in 1 PPM increments, until a signal is found. So far it seems to be working well.
Hey @kkotowick, since it's your change, want to make a PR to this project with the auto-tune functionality? Very useful to have for newcomers trying to calibrate their SDRs.
It wouldn't be a bad bit of code to slap into @robotastic 's Trunk Recorder.
I originally had the variable name of "ChaseControlFreq" in mind for something like this, but that's just minor stuff.
This would be useful even on the HackRF. As the HackRF is stupidly sensitive to temperature changes. If the room temp is at or near 80F, the PPM 'drift' sits right around 15 (at 15.2 currently). In the low 70's, it's around 12-13 PPM. At a cold start, the drift quickly goes from 0 to 5 to 12 in minutes. Having to baby the drift, from RTLSDR's to a HackRF is timesome. :)
@EricTendian I made those modifications for a specific application, but if I remember correctly it was kind of hacked together for a specific system and not fully generalized. If you want to go ahead and clean it up a bit, do some testing, generally make it production-worthy, etc., then feel free to submit the pull request. I don't need the credit ;)
Have applied the added code from https://github.com/robotastic/trunk-recorder/commit/3584e844815b052546e6985baf106380e7ad7cb2 to the latest code from @robotastic 's repo.
Can confirm this code is functioning without issue. Waiting for the drifting to kick in and see how well it functions. I have the PPM drift set to increments of 0.1, with a range of 20 right now. Seems to work!
Will attempt to make some diff files for this later this weekend. As the code locations where the new code is added at, is a bit different from the lines shown.
Cool - let me know how this goes. If you post a fork, I can try it with my HackRF which drifts a bit.
@robotastic looks like the code was posted here: https://github.com/robotastic/trunk-recorder/tree/3584e844815b052546e6985baf106380e7ad7cb2
Look forward to getting this working as well, I'm following now, so I can integrate it into the new hardware I'm starting to push out.
So after testing this code, it does not work well on a HackRF with a dual or multi-system configuration CC setup, this is with a 10MHz and 12.5MHz bandwidth setting. From my findings, the code works "OK" for a single P25 system setup.
To explain, if one CC's system detects a PPM drift, it will +# the value you've set at ppm_adjust_interval. However, the other system, in the same frequency range will then subtract the PPM drift that got applied within the second the PPM positive drift correction occurs. This in turn makes TrunkRecorder a bit unhappy with audio output, and audio gets a little.... "expanded" in sound. I am unsure how to accommodate the drift checking on a source basis. Looking deeper, it may be wise to include an ignoreDriftCheck boolean on a per system basis (this likely wouldn't be a problem on a multi-RTL-SDR 2.4MHz bandwidth spanned setup).
The other issue i've found is if the PPM is set too low or too high - with a modestly high "PPM search range", and the PPM drift corrector begins its scanning, it will sometimes lock onto another nearby CC frequency that is not of the system that you may desire to monitor. Am unsure if @robotastic 's TrunkRecorder has code in place like what @treehouseman's repo has in regards to system details.
Another issue, which is a more long-standing issue, is that if the SDR/OsmoSDR based device doesn't disconnect or exit gracefully, and if you go to restart ./recorder, the PPM adjuster goes haywire and will scan up and down per given configuration until it fails. May be worthwhile putting in a on-execution check to see if the RTL device isn't locked/in-use.
Attached is some audio output when a PPM drift occurs. At first, i thought it was encryption or digital noise bleed-in, but this is what it does when the PPM drift correction activates with a HackRF.
TLDR: The Code works, but it's not too graceful when a PPM Drift Correction occurs. :(
Relooking at this issue, and troubleshooting some of my own problems.. I've noticed that SDRTrunk for unix is doing this quite well -- so the code has to be available to do this pretty easy. Now SDRTrunk is in Java.
For a long time, my RTL sticks had no drift, but now, I'm starting to get variable drift on most of my SDR's that have been running for 2 years without any error correction, and its painful to try and "figure out" what the proper error or PPM should be. I've seen other comments by @robotastic that ppm settings may not be implemented well, so getting the exact hz of error is difficult. Any suggestions on determining this on a console only (no xwin) system?
@kcwebby I haven't used this script in a while because of the auto-tune code mentioned above, but previously I was using a script I found by someone with the same issue which uses a reference frequency and rtl_power to tune a RTL-SDR's ppm to match the peak of the transmission: https://github.com/CrimeIsDown/ppm-calibrate (console-based)
I haven't used that code in quite some time so can't guarantee it will work properly, but that's a general idea to try out.
Awesome - glad you have had success with it. Wish we could get Luke or Kyle to merge this into the new code. Glad its working well for you!
Luke? I put it in a pull request, if you can take a look at it.
On Sun, Sep 29, 2019 at 10:40 PM Eric Tendian [email protected] wrote:
@kcwebby https://github.com/kcwebby I haven't used this script in a while because of the auto-tune code mentioned above, but previously I was using a script I found by someone with the same issue which uses a reference frequency and rtl_power to tune a RTL-SDR's ppm to match the peak of the transmission: https://github.com/CrimeIsDown/ppm-calibrate (console-based)
I haven't used that code in quite some time so can't guarantee it will work properly, but that's a general idea to try out.
— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/robotastic/trunk-recorder/issues/217?email_source=notifications&email_token=ACSNSAVGFR24SHZTWWJOGHTQMFYKJA5CNFSM4GPSK5CKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD74KELQ#issuecomment-536388142, or mute the thread https://github.com/notifications/unsubscribe-auth/ACSNSAXQNORDL53ZXQ3DENTQMFYKJANCNFSM4GPSK5CA .
The auto frequency error correction seems to work well in SDRTrunk, it may be worth looking at how it was done there. Finding and setting the error in trunk-recorder for each SDR is definitely one of the biggest hurdles for new users and a pain for people with SDRs that have some drift and/or a large number of SDRs. See https://github.com/DSheirer/sdrtrunk/blob/master/src/main/java/io/github/dsheirer/source/tuner/FrequencyErrorCorrectionManager.java
