Extending simulator to support .pcap tracing

[non-det-alle]

.pcap tracing (Issue #87):

• helper/lora-helper: Extending class to support promiscuous traffic sniffing using .pcap files. Implementation follows the one in PointToPointHelper base Ns-3 class. Check EnablePcap() Ns-3 docs for usage;
• model/lora-net-device: Adding .pcap tracing in the gateways' net devices by adding the source callback into Send and Receive functions.

Other fixes:

• model/end-device-lorawan-mac.h: Changing dimension of FCnt header field from 1 octet to 2 according to specifications;
• model/end-device-lorawan-mac.cc: Moving check for the max allowed MACPayload after frame header addition, according to specifications.

EDIT: The two minor fixes moved to other pull request because unrelated

[DvdMgr]

This looks like a great start to implement pcap tracing, thank you for this contribution! I have a couple of questions:

• I had to add DLT_LORATAP = 270 as a DataLinkType in src/network/trace-helper.h. Are there any additional changes to the ns-3 codebase that are required to support this?
• I tested this by calling EnablePcap in complete-network-example.cc, and the resulting file did indeed contain LoraTap packets. However, it seems like a lot of the information is missing - there is no info about the rssi, the sf, source and destination addresses, and so on (which I expected to find based on the LoRaTap definition). Is this expected or am I missing something?

[non-det-alle]

Hello @DvdMgr.

• About the first point, I did this sometime ago and so I eventually forgot that I had added DLT_LORATAP as a DataLinkType. Sorry for not mentioning it. To my knowledge this is the only change needed to start working with the DLT_LORATAP .pcap format. Do you have ideas on how this could be managed once the implementation is good enough to be merged? Maybe by adding this information in the module instructions?
• This early implementation of .pcap tracing was useful to me to inspect the content of the MAC header, even if it didn't fit the LoRaTap definition, so I thought it might be a good thing to open a pull request. Your observation makes me realize it may be still too incomplete. I will continue working on this so that it fits the exact format, should I close the pull request for the time being?

As you can see I also added two small fixes. We already discussed the dimension of FCnt via email some time ago, but what do you think about the second one? The regional parameters document (RP002) says that the byte limitation as is defined in the code refers to the entirety of MACPayload field, so I moved the check after LoraFrameHeader is added to any packet.

[DvdMgr]

As for the change to the DataLinkType, I don't see why they wouldn't accept a merge request for this very small change directly in the ns-3 repository! This would ensure our module keeps working with the 'vanilla' ns-3 repository - asking users to change files in ns-3 and providing something that doesn't work out of the box can get messy...

If you have some spare time to keep working on the .pcap implementation, I think the best way to proceed is to keep adding to this pull request (so that I and other can also follow progress and contribute) and merge it once we are satisfied with the overall result.

As for the check on maximum packet size, nice find! I agree that it should take into account the FrameHeader too, since it's defined for the PHY payload.

[non-det-alle]

Hello,

I implemented a new header with the necessary fields for .pcap traces to be displayed correctly and moved the small fixes to another pull.

I filled the header using the info from LoraTag. The content of some fields in the new header are still missing (SNR, RSSI related), let me know if you have ideas on how to fill them. Also it is not super easy to understand how this info is encoded, my final interpretation is that something like (dBm value is -139 + rssi) in the documentation means that here rssi is the uint8_t value and this is an explanation of how it will be displayed.

Design remark: I would have liked to be able to detect from inside LoraNetDevice if the traced callback was hooked to a pcap sink, but after investigating the issue it doesn't seem possible, so I settled to adding a flag in the devices activated by EnablePcapInternal() in LoraHelper.

[renzoe]

HI @non-det-alle !! Incredible , thank you for this contribution, I "manually" (...) merged it on my local repo. I can help you to polish all this. I will use it mostly on End Device-side. As thing stands, the ED does not capture anything, because I think the functions where the traces are fed are re-defined by end-devices. In short, I had to add the m_promiscSnifferTrace(packet); at different places. To be precise:

• ClassAEndDeviceLorawanMac::SendToPhy
• ClassAEndDeviceLorawanMac::Receive [I did it only if the message is for us.. so it is actually not promiscuous mode :S]

I think we have not so trivial design decisions to make. In order to impact the less possible amount of core-code. And feed the trace cleanly. At lora-phy all devices could share the code. But will be only promiscuous. In case of a node wanting to capture "unicast" received packets, the appropriate layer to implement the capture is at the "lower" MAC (other example, for sending: after you checked that you respect the duty-cycle restrictions and you will actually modulate this frame). See: https://github.com/signetlabdei/lorawan/pull/103/commits/7e06b92cd3e1c8991972249e99848ec45694b192#diff-00db5d5c0bd2658d8a5fbcbf25d6c0cb598b766b44c02281e5717b044205f8b5R124

In any case, thank you @non-det-alle. I will contact you privately, and we may work together on this, or publicly but in your private fork. In any case, I got this working in one day thanks to your contribution :).

[non-det-alle]

Hello @renzoe, I agree with you that the m_promiscSnifferTrace(packet); trace source for downlink packets should be fired at a lower level. The problem is that Ns-3 existing Pcap tracing system is designed for NetDevices (see) and it wouldn't hook the trace sink correctly in other classes (did you actually manage to do this?). To do this we would need to create a new Pcap helper (more specifically a whole new trace-helper) tailored to our needs.

[renzoe]

HI @non-det-alle ! Thanks for the reply. "Short" answer:

The problem is that Ns-3 existing Pcap tracing system is designed for NetDevices (see) and it wouldn't hook the trace sink correctly in other classes (did you actually manage to do this?).

Yes, I managed, it was you that showed me the way. See how they do it a ns-3 WifiHelper: wifi-helper.cc Once you have your NetDevice, you ask for his PHY (or MAC in my case), and then you hook. Of course if the device has no PHY there is nothing to sniff :P.

In my dirty way, because I put it at MAC, I am asking for the MAC layer of the Net Device, and finally you have something like this (LoraHelper::EnablePcapInternal of lora-helper.cc) pcapHelper.HookDefaultSink<LorawanMac> (mac, "PromiscSniffer", file);

Actually, having our own trace-helper is not a bad Idea... we can inherit from the existing one, and also add/redefine the DataLinkType enum to include DLT_LORATAP = 270; problem solved to not depend on the dev branch of ns-3 (still to see if the merge this pull request that will include DLT_LORATAP).

[non-det-alle]

Hello, This is to let you know that DLT_LORATAP has been officially added to the main Ns-3 codebase as one of the possible Pcap filetypes. Rejoice!

[asmanuha]

Dear All, Thanks for your effort to provide Pcap application so that we can see what's going on in the data transmission. However After I successfully implement Pcap trace in example file "network-server-example.cc" , I found out that there are NO DATA captured in the tracer. I compared the Pcap result gathering from - pointToPoint.EnablePcapAll("first") - in first.cc, and with this lorawan scenario. Could you advise me how to integrate pcap application, so that the LoraWan DATA appear on Pcap tracer? (For example data send via oneShotHelper or PeriodicSenderHelper). My goal is that we can able to compare encrypted and un-encrypted lorawan data as a bridge to implement security on LoRaWAN.

Thanks

[DvdMgr]

Hey @non-det-alle, @renzoe - can you provide a quick summary of the current state of this pull request? The branch linked to this PR does not seem to compile, is there a more updated version somewhere that I can test?

[non-det-alle]

Hello @DvdMgr,

It's been a bit since I have taken a look so I don't remember, I'll go over it in the weekend and get back to you so we can decide what to do with it.

In the meantime, have you tried it with the latest ns-3-dev? The support for LoRa Pcap files was merged some months ago but may still not be in an official release.

[non-det-alle]

After I successfully implement Pcap trace in example file "network-server-example.cc" , I found out that there are NO DATA captured in the tracer. I compared the Pcap result gathering from - pointToPoint.EnablePcapAll("first") - in first.cc, and with this lorawan scenario. Could you advise me how to integrate pcap application, so that the LoraWan DATA appear on Pcap tracer? (For example data send via oneShotHelper or PeriodicSenderHelper).

@asmanuha sorry, I completely lost your message. I'll check the implementation and get back to you.

[DvdMgr]

have you tried it with the latest ns-3-dev?

Yes, it looks like compilation fails because m_promiscSnifferTrace and LoraHelper::EnablePcapInternal are not declared (I suspect this could be fixed by having LoraNetDevice and LoraHelper extend the appropriate classes).

[non-det-alle]

Hello all,

I restored the code that enables Pcap tracing. I had accidentally overwritten the Pcap implementation in the last fix merge, I am truly sorry .

I left a commented-out line to enable tracing in complete-network-example.cc for you to test it. I'll try to continue the work and address the discussion with @renzoe about where is the best place to sniff the packets. As he suggested, I'll follow at the implementation in wifi-helper.

[non-det-alle]

Hello @DvdMgr and all,

I think the Pcap tracing implementation can be now considered complete. I have the following comments on the last two commits so that we can discuss:

• Implementation takes inspiration from the one of WifiHelper and now sniffing is done at physical level (LoraPhy class and children). In WifiHelper and related classes, both received and sent packets are added to the trace of Pcap enables devices. This is because real Wifi devices can operate in monitor mode to sniff frames being received and transmitted. That's not the case for LoRa. Still, we know that a LoRa device can be used to sniff packets on the spectrum (see). On top of this, LORATAP Pcap header includes measurements taken for received packets (RSSI, SNR). So I decided to limit sniffing for devices (both end devices and gateway) to just received packets. As a consequence, this implementation is not an abstraction to trace all frames in the simulation. It just allows to turn on .pcap tracing of the received packets in the desired devices, producing a file for each device we want to use as a sniffer.

• One of the fields of the LORATAP header is SNR. This needs to be discussed, but for the time being I added a method in LoraInterferenceHelper that is very similar to IsDestroyedByInterference but only computes the total SNR using the cumulative interference energy of overlapping packets. This adds some code duplication and entails double execution of a not-so-light procedure. Initially I had integrated the SNR computation in IsDestroyedByInterference by attaching the SNR to the returned value with a tuple, but changing the return values was breaking tests and I didn't want to mess with that. I also had to add a field to 'LoraTag'. Finally, when there are no interferers the SNR would be infinite, so I capped the value to 10dB (see).

Let me know if something doesn't work, is missing or you think should be done differently, I'll be on it asap.

EDIT: I forgot to mention that I reverted my modification to complete-network-example.cc and added pcap-example.cc, which is the same example with Pcap enabled by default. Mind that it will produce a file every time.

[DvdMgr]

@non-det-alle, thanks for the prompt response! I have tested the PR with Wireshark and it seems to work as expected, thanks for the stellar job.

I have the following comments:

1. Need to add pcap-example.cc to examples/wscript.
2. I'm fine with the PR only enabling tracking of received packets, for now. I gave it a quick look, and did not find the requirement for tracing both sent and received packets in the pcap related ns-3 classes. As long as we properly document this, we should be fine.
3. I tried enabling confirmed traffic, and told the LoraHelper to also sniff packets coming from endDevices. This works smoothly, and a .pcap file is created for each ED as expected. When I open an ED's .pcap file, however, the payload portion of the DL message shows up as corrupt in Wireshark (the LoraTap header and the LoRaWAN protocol fields are correct). I guess this is because of the empty payload - we might need to take a look at the Wireshark dissector to figure out why it's giving an error, however I'd say this is only a minor issue.
4. Since we support a couple of MAC commands, I'd like to test whether they are captured and parsed correctly by Wireshark as well, but I didn't have time to do that yet.
5. On SNR: from what I understand, the value that is currently computed is closer to a Signal to Interference Ratio (SIR), not considering background noise. I think we should take this component into account - this way, we could expect that in the absence of interferers SNR will drop as the RSSI drops (as in Figure 5.1 of the document you linked). I think inverting Eqn. 1 from this document could be a good starting point, where instead of S we have the RSSI - in addition to this, we should consider the presence of interferers as in the current implementation.

Let me know what you think about the points above; I can spend some time working on the code if you need a hand!

[non-det-alle]

Hello @DvdMgr,

Glad you like it! My comments:

2. I'm fine with the PR only enabling tracking of received packets, for now. [...]

Pcap related classes leave freedom on where and how the packet collection happens. The only constraint is that you need to add an header with the fields required by TAP formats, if not already present. It wouldn't be difficult to implement workarounds to collect sent packets. But I choose not to, because in other modules they try to keep it as realistic as possible (that's why I mentioned the Wifi example).

3. I tried enabling confirmed traffic [...]

I'll try to test that too. In theory, the packet is written as it is after the TAP header.

4. Since we support a couple of MAC commands [...]

I think that the added value of Pcap tracing is that it could be very useful for testing future features too.

5. On SNR [...]

As you may have noticed, I don't have a strong background in radio. That said, I can follow your directions on that! I think your idea is neat, I can try to implement it (I think the interference helper is still the right place, do you agree?). How would you introduce interferers in the SNR formula you mention?

Let me know. It shouldn't be a long task, I'll let you know if I need help with the code.

[non-det-alle]

Hello @DvdMgr,

Sorry for the long pause. I addressed your comment on SNR.

[non-det-alle]

I checked the malformed payload error for downlink frames sniffed by devices. After a bit of testing with LinkAdrReq frames, it seems that the Wireshark dissector expects 4 bytes of MIC at the end. This is in line with specifications, I am not sure how this should be addressed. The issue is present in uplink packets too, where the trailing 4 bytes of the payload are interpreted as MIC.

You can test by adding status->m_reply.payload = Create<Packet> (4); at line 156 of adr-component.cc and activating devices pcap in ADR example. Remember to lower the number of devices to avoid filling up the folder with traces.

[brunocitoni]

Sorry to butt in, but I recently opened an issue (123) about the same MIC problem https://github.com/signetlabdei/lorawan/issues/123, which is giving rise to slightly incorrect Time-on-Air for packets so I am interested in seeing how this develops!

[renzoe]

Hello @DvdMgr,

Sorry for the long pause. I addressed your comment on SNR. Thank you very much @non-det-alle ; I am more than sorry for my absence too. I changed works in Aug 2021, and did not have time to continue with this at all. This summer I will have some time. I will be open to discuss/lend a hand. Good catch @brunocitoni about the MIC !

[non-det-alle]

Hello,

As mentioned previously by @DvdMgr, I added bidirectional Pcap sniffing (before was only on reception). For packets captured on transmission I put 0 for SNR and RSSI in the respective mandatory Pcap fields, as the sender does not have this information (the LORA_TAP format was designed for sniffing on reception).

Also I improved PcapExample to provide traces of a more interesting scenario. Here, a single device is sending uplinks to a single gateway, and ADR is enabled. With default parameters, for the first 4 frames the device is duty-cycle limited at SF12. Then, after receiving an ADR reconfiguration from the server to SF9, it resumes its 60 seconds periodicity.

I added a second commit with a simple fix to malformed frames: a fake MIC is added at the end of packets before they are sent (both uplink and downlink).

[non-det-alle]

I am closing this branch due to lack of replies from maintainers.