Talkgroup Numbering

[leee]

Currently, trunk-recorder (as well as OpenMHz) tries to operate on the common ground of talkgroups being in decimal and ignores hexadecimal talkgroup values. This isn't a poor choice, since it is my understanding that the various scanner websites (and by extension, the scanner hardware they support) operate in decimal as well.

This works, so long as everything stays on the same page. Pages include:

• trunk-recorder writes to files prefixed with talkgroup in decimal
• talkgroups are handled on OpenMHz in decimal
• talkgroup csvs for OpenMHz are handled in decimal only as well (and intentionally requires that you remove the hexadecimal talkgroup column, occasionally a pain point seen in issues like #266)

In theory, local talkgroup csvs for recorders should be handled in decimal as well, but you occasionally see things not fall in lockstep like in #306 which affects P25 systems:

After digging around, folks on gitter chat concluded that the HEX value in the CSV has to be filled out properly for the priority to function.

---

Type II trunks make this hex vs dec juggling worse. Hexadecimal talkgroups are joined with a status nibble in the control channel stream. So instead of "TG $ABC", you see "Data field $ABCD", where $D is that status nibble whose value tells us if a talkgroup is encrypted, if there's an emergency call on it, if it's a patch or msel, if it's an ATG, etc. We shouldn't be logging calls like this, which should be handled by just talkgroup and not talkgroup with status.

This is compounded by the scanner website data. Such data is always saved with decimal talkgroups always having 0 status. Calls with a non-0 status nibble do not get matched to these known talkgroups.

Now that we are aware of encrypted calls and emergency calls with #388, we need to start thinking about how to fix this issue so non-0 status talkgroups don't get mishandled like they are right now.

---

As we talk about this though, I find it important to bring up the fact that trunks frequently get upgraded piece by piece, and talkgroups remain the same as legacy infrastructure gets homed on P25 infrastructure. That means a talkgroup file for a Type II trunk is also good for the P25 trunk replacing it.

However, we currently would not be able to use the same csv for both a Type II trunk and a P25 trunk. Since we try to operate on decimal, and Type II trunks include the status nibble, decimal talkgroups do not line up and are bit-shifted by 4 thanks to that status nibble! We frequently see this "Type II scanner land decimal talkgroup" to "actual decimal talkgroup, also used by P25" as a multiple of 16.

What does line up though are hex talkgroups. For some reason, the scanner websites have deemed it useful for Type II listings to have decimal and hexadecimal talkgroup columns, but the talkgroup columns don't retain a status nibble, whereas decimal columns do.

---

Since we'd have a breaking change trying to fix Type II talkgroup numbering, we may as well take advantage of the timing and make everything operate on hexadecimal talkgroup numbering. In order to do this though, several things will have to happen:

1. ensure popular upload targets like OpenMHz are prepared for hexadecimal talkgroup formatted audio and json file names - this could be by including a v2 or all hex somewhere on upload, or similar notification.
2. think about users in this change. Users can come in several flavors for this change, and I'll discuss that below.
3. make all parts of trunk-recorder operate on hexadecimal talkgroups and upload targets be updated with a v2 flag.
4. include this change only in a major release where users are hopefully well informed that they need to double check their config and talkgroup files.

Regarding users: they can come in the following flavors:

1. local talkgroup files with hex columns populated correctly. All they have to do is update their targets, especially ones like OpenMHz which required stripping that column before.
2. local talkgroup files with hex columns not populated. We can alert but continue to operate as if we had no local TG file. The hope is that any messaging and notices upon updates is what alerts these users to update their files.
3. local talkgroup files with hex columns filled with garbage data. The easy way out would be to respond in kind: garbage in means garbage out.

It's possible that this change could involve a configuration setting for operating in hexadecimal or decimal (but which kind, real or scanner-land for Type II systems?) but the more I think about it, the more I feel that it's easiest to rip out the pain point in one go than apply bandages that will need to be handled eventually, which is just kicking the problem down the road.

[robotastic]

I am a little confused, but I think that is because I am coming at this as a programmer. Internally, all of the Talkgroup numbers are stored as an integer. From the SDR they come in as a series of bits/bytes that get converted into integers. When you print them out the TG number you can choose whether they are in Hex or Decimal, but that is just for the human. Same thing for taking input from a file/command line. It is just a matter of how you parse that input string.

For this:

In theory, local talkgroup csvs for recorders should be handled in decimal as well, but you occasionally see things not fall in lockstep like in #306 which affects P25 systems

That is just because the parser for the CSV file is brittle. The hex value is not used for anything, it is thrown away. In this case, the CSV parser just needs something that could be a hex value or it will fail.

That part I am confused on is the Smartnet II talkgroup representation. I am correct that TR is including the Status Nibble as part of the TG address? I agree, we shouldn't be including the TG status as part of the TG address. If so, it sounds like there is a bug in the way Smartnet messages are being parsed and the TG address converted into an integer. Let's fix that.

Do we just need to switch the lines where message.talkgroup is set to full_address to be address? https://github.com/robotastic/trunk-recorder/blob/master/trunk-recorder/systems/smartnet_parser.cc#L240 I believe address is just full_address with the status bits set to zero.

Whether the talkgroup number gets displayed to humans as a base 10 or base 16 should be a different issue ticket.

[leee]

I am a little confused, but I think that is because I am coming at this as a programmer. Internally, all of the Talkgroup numbers are stored as an integer. From the SDR they come in as a series of bits/bytes that get converted into integers.

Those bits/bytes are handled as integers, no converting.

When you print them out the TG number you can choose whether they are in Hex or Decimal, but that is just for the human. Same thing for taking input from a file/command line. It is just a matter of how you parse that input string.

Right, and printing integers regardless of if they were introduced as hexadecimal or (actual) decimal is a matter of string handling.

---

That part I am confused on is the Smartnet II talkgroup representation. I am correct that TR is including the Status Nibble as part of the TG address?

Yes.

I agree, we shouldn't be including the TG status as part of the TG address. If so, it sounds like there is a bug in the way Smartnet messages are being parsed and the TG address converted into an integer.

The "bug" is not within the way data is being "converted", but yes, parsing should only take full_address and know that the first three digits are the talkgroup. Whether or not a status nibble of 0 is included or not is at question.

Do we just need to switch the lines where message.talkgroup is set to full_address to be address? https://github.com/robotastic/trunk-recorder/blob/master/trunk-recorder/systems/smartnet_parser.cc#L240 I believe address is just full_address with the status bits set to zero.

Your assessment is correct, and using talkgroups with status nibble of 0 regardless of actual status is what scanner websites have for Type II systems decimal talkgroup values.

Doing this is a quick fix, and would serve the purpose of making scanner website Type II decimal talkgroup data usable again regardless of non-zero status bit calls.

Users (and upload targets) who have had to craft custom talkgroup data files to account for status bits will either need to reform them or can obtain a fresh non-edited copy.

What I am saying is since users will see this change that may break things involving their talkgroup data files, might as well affect change all the way.

---

Whether the talkgroup number gets displayed to humans as a base 10 or base 16 should be a different issue ticket.

This question is better reframed as:

whether the talkgroup number gets displayed to humans as

• "real" base 10 (applies to both P25 and TII, currently P25 does this)
• "scanner" base 10 (applies only to TII, currently TII does this)
• base 16 (applies to both P25 and TII, and scanner website data supports base 16 for both system types "natively")

should be a different ticket.

For the two different system types:

P25: (e.g. talkgroup $a01)

• can use "real" base 10 (2561) with no problem as scanner website data has this column properly
• "scanner" base 10 does not apply
• can use base 16 (a01) with no problem as scanner website data has this column properly

TII: (e.g. talkgroup $a01 with status nibble say $a)

• can NOT use "real" base 10 (2561) as scanner website data has this column as "scanner" base 10
• can use "scanner" base 10 (40976, which considers the status nibble as 0) with no problem as scanner website data has this column (improperly)
• can use base 16 (a01) with no problem as scanner website data has this column properly

[robotastic]

Cool - I think I have a better understanding. So the big part that I was missing is that the Dec column listing on Radio Reference is not simply a Hex to Decimal conversion for SmartNet type systems. Luckily, TR has been using the address + status nibble for SmartNet addressing already. It just hasn't been zeroing out those status-bits, so if a system had those status bits set, you would have a Talkgroup mismatch.

The "bug" is not within the way data is being "converted", but yes, parsing should only take full_address and know that the first three digits are the talkgroup. Whether or not a status nibble of 0 is included or not is at question.

We should be using the address that is already in the code, which zeros out the 4 status bits:

long address = full_address & 0xFFF0;

https://github.com/robotastic/trunk-recorder/blob/master/trunk-recorder/systems/smartnet_parser.cc#L136

I just created an address branch with this change, and it works fine on my local SmartNet system.

I can create a Note at the top of the readme for folks running SmartNet systems with customized TG Numbers.

[leee]

I just reread my own comment, my technical writing skills really took a day off this morning. You got to the right assessment though!

Yep, address will work if you decide to continue using scanner-land decimal. I still firmly believe things need to move to hexadecimal numbering, but this quick fix can be adopted for now and that discussion be sussed out another time.

Your address branch commit looks fine and I was running the same exact change a long time ago. It is good for inclusion into master.

[ginomiglio]

This was the top issue I was having with trunk-recorder. Agree that the numbering system is a mess but I'm happy there was a quick fix for this particular issue. I'm now running the address branch on my local smartnet system, and it completely fixed the problems I was having (fire department dispatches on multiple TGs and encrypted TGs would end up with TGIDs that didn't match the listed ones, which made them a pain to keep track of).

@robotastic any plans to merge that change? So far it's a big improvement!

[ginomiglio]

Rebased your branch and opened a PR. Thanks for fixing this, y'all.

(edit: I removed the "fixes" from the description in case you want to keep this issue open for the longer term convo)

[robotastic]

Thanks @ginomiglio - I think I will add this into an upcoming release I am putting together based on #394. It will be a breaking change and people will need to update their config files... good time to also fix csv files. This will atleast fix things in the short term. We can figure out the best way to harmonize numbering after that.

[rosecitytransit]

There's a good description of talkgroup numbering on Wikipedia https://en.wikipedia.org/wiki/Motorola_Type_II#Status_bits