Packets dropped on mt7615 for A-MSDU transmissions

[drandyhaas]

Hi, In trying to figure out what's going on in issue 387: https://github.com/openwrt/mt76/issues/387 I found that packets are dropped by the mt7615 driver (at least on my winstars 538a6) when sent from another router as an A-MSDU aggregated packet. Large pings (with -s 2000 for instance) are replied to by the mt7615 router when sent as "fragmented ip" packets, but not when sent as "a-msdu" packets. See wireshark sniff below (from a 3rd machine monitoring traffic in the middle).

Is A-MSDU (and A-MPDU) supported by mt76? At least for reception (rx)? The driver advertises that it is (in "iw phy0 info").

[nbd168]

A-MSDU itself is supported. Could you please send me a .pcap of a monitor mode capture that shows the issue?

[drandyhaas]

Here is a pcap...

ath2mt.pcap.zip

[nbd168]

What kind of router and firmware are you using on the A-MSDU transmitter side?

[drandyhaas]

On the transmitter side it's a linksys ea8300, running openwrt 19.07.3, with ath10k-ct drivers. From the dmesg: "ath10k 4.19 driver, optimized for CT firmware"

[nbd168]

Are you using 802.11s? Or is this a normal 4-address mode (WDS) link?

[drandyhaas]

It's 802.11s. No encryption (for testing purposes).

[nbd168]

I just spent some time reading the mac80211 A-MSDU parsing code and the 802.11 standard. If I'm reading things correctly, it seems that there may be a 802.11s specific bug in the mac80211 A-MSDU decap code. I'll work on a patch for you to test.

[drandyhaas]

Ok, I look forward to giving it a try. Thanks!

[drandyhaas]

Anything else I can do to help here? Maybe point me where to look in the code?

[nbd168]

The problem is that the bug comes from a wrong assumptions in the rx processing code. The code assumes that the mesh header comes after the 802.11 header before A-MSDU subframes, when in fact the mesh header is part of each individual subframe. Fixing this flawed assumption will require some significant rework. I will work on it, but I need more time to get it done.

[drandyhaas]

Ok, thanks! Just let me know if there's anything else I can do to help.

[drandyhaas]

Any progress?

[drandyhaas]

Is there some estimate of the time this could take to fix? If it's going to be long I'll have to try to hack it myself.. Thanks.

[drandyhaas]

Can you please at least point me to the place in the code where this is done? As far as I see, the 802.11s decap is all done in the standard Linux code, so I'm not sure where to look in the mt76 code. Thanks.

[nbd168]

The issue is in the Linux stack rx code, not mt76. ieee80211_data_to_8023_exthdr in net/wireless/util.c assumes that in mesh mode, the mesh header immediately follows the 802.11 header and applies to all subframes. According to the 802.11-2016 standard, chapter 9.3.2.2.2, figure 9-56, the mesh header is part of each A-MSDU subframe. Taking this into account will need some significant rework of the rx path and cfg80211 utility functions, including the mesh forwarding code. The A-MSDU decap code needs to be adjusted to leave the mesh header in place, and mesh forwarding needs to happen later in the rx processing chain.

[drandyhaas]

Thanks very much for the info. What confuses me here is how other drivers, like ath10k etc manage to work just fine, if the problem is upstream in a piece of shared linux decap code. I guess mt76 is the only ones using this piece of linux code?

[nbd168]

ath10k has hardware A-MSDU decapsulation, so it bypasses the software part.

[drandyhaas]

I see. Does anyone have the datahseet (in English prefereably) of the MT7615? I'd be surprised if it doesn't have hardware offload capabilities too.

[nbd168]

MTK told me it can't offload Mesh A-MSDU.

[drandyhaas]

OK. Note that mesh forwarding at the 802.11s level is not so important to get right for now. Usually people use BATMAN to do forwarding at layer 2 ethernet, and explicitly turn off mesh forwarding. So it may be possible to get the basic mesh a-msdu working fine for now without getting mesh forwarding working.

[drandyhaas]

Any hope of finding time to work on this? I could try, but it would be challenging...

[drandyhaas]

Until a real fix for this is available, it works OK to simply disable mstu. I comment out this line ieee80211_hw_set(hw, SUPPORTS_AMSDU_IN_AMPDU); here https://github.com/openwrt/mt76/blob/master/mac80211.c#L307 and I get reasonable performance of about 150Mb/s in both directions over 80211s.

[dhewg]

Until a real fix for this is available, it works OK to simply disable mstu. I comment out this line ieee80211_hw_set(hw, SUPPORTS_AMSDU_IN_AMPDU); here https://github.com/openwrt/mt76/blob/master/mac80211.c#L307 and I get reasonable performance of about 150Mb/s in both directions over 80211s.

Thanks for that!

If anyone cares, here's a patch to apply to openwrt to do exactly that: 0001-mt76-HACK-disable-A-MSDU.patch.txt

Without patch:

[  5] local 192.168.0.2 port 5201 connected to 192.168.0.1 port 41296
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec   501 KBytes  4.10 Mbits/sec                  
[  5]   1.00-2.01   sec   600 KBytes  4.88 Mbits/sec                  
[  5]   2.01-3.01   sec   679 KBytes  5.56 Mbits/sec                  
[  5]   3.01-4.01   sec   455 KBytes  3.73 Mbits/sec                  
[  5]   4.01-5.00   sec   513 KBytes  4.23 Mbits/sec                  
[  5]   5.00-6.00   sec   492 KBytes  4.03 Mbits/sec                  
[  5]   6.00-7.00   sec   547 KBytes  4.48 Mbits/sec                  
[  5]   7.00-8.01   sec   615 KBytes  5.00 Mbits/sec                  
[  5]   8.01-9.01   sec   496 KBytes  4.07 Mbits/sec                  
[  5]   9.01-10.01  sec   532 KBytes  4.36 Mbits/sec                  
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-10.01  sec  5.30 MBytes  4.45 Mbits/sec                  receiver

With patch:

[  5] local 192.168.0.2 port 5201 connected to 192.168.0.1 port 41300
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec  5.44 MBytes  45.5 Mbits/sec                  
[  5]   1.00-2.00   sec  6.12 MBytes  51.4 Mbits/sec                  
[  5]   2.00-3.00   sec  7.08 MBytes  59.4 Mbits/sec                  
[  5]   3.00-4.00   sec  4.75 MBytes  39.8 Mbits/sec                  
[  5]   4.00-5.00   sec  7.24 MBytes  60.7 Mbits/sec                  
[  5]   5.00-6.00   sec  7.38 MBytes  62.0 Mbits/sec                  
[  5]   6.00-7.00   sec  7.23 MBytes  60.7 Mbits/sec                  
[  5]   7.00-8.00   sec  7.50 MBytes  62.9 Mbits/sec                  
[  5]   8.00-9.00   sec  7.65 MBytes  64.2 Mbits/sec                  
[  5]   9.00-10.00  sec  7.89 MBytes  66.2 Mbits/sec                  
[  5]  10.00-10.07  sec   557 KBytes  69.8 Mbits/sec                  
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-10.07  sec  68.8 MBytes  57.4 Mbits/sec                  receiver

[emmarrem]

Hi, I am a noob and I do not know how to patch the openwrt firmware or comment out the line as mentioned above. Could you please shed some light on how this can be done. I have also been facing the same speed issues by using 802.11s.

[dhewg]

@blocktrron came up with a cleaner way to work around the issue, attached a patch for the openwrt tree: 0001-mac80211-mask-nested-A-MSDU-support-for-mesh.patch.txt I'm successfully running this on a mesh setup with one mt76 enabled host and one without it since july.

[nscnd]

Hi there @nbd168,

I've been hitting this issue with mt7915e driver on a MT7916 chipset. Did you find time to tackle this in proper way yet ? Or this there no other way than disabling AMSDU ?

Thanks

[Djfe]

He's working on it 🥳 (edit: and has been for some time, but I only picked up on this issue recently) https://www.spinics.net/lists/linux-wireless/msg232408.html

[shown19]

Hi, I am a noob and I do not know how to patch the openwrt firmware or comment out the line as mentioned above. Could you please shed some light on how this can be done. I have also been facing the same speed issues by using 802.11s.

Hi may I know, if you managed to patch yours? I wanted to try patching the mac80211.c also in hope that this will fix the packet loss and large ping issue in my MT7621 Chipset, device: Newifi D2. I already watched tutorial in Youtube but only in how to build the firmware not the applying the patch method. Would be glad if someone could help me, I'm willing to learn and be guided. Thank you.

[sayap]

@shown19 can try this:

cd /path/to/openwrt/source/
wget https://github.com/openwrt/mt76/files/9661690/0001-mac80211-mask-nested-A-MSDU-support-for-mesh.patch.txt -O package/kernel/mt76/patches/0001-mac80211-mask-nested-A-MSDU-support-for-mesh.patch

This will place the patch shared above under package/kernel/mt76/patches/. You can then run make to build the firmware again.

[shown19]

@shown19 can try this:

cd /path/to/openwrt/source/
wget https://github.com/openwrt/mt76/files/9661690/0001-mac80211-mask-nested-A-MSDU-support-for-mesh.patch.txt -O package/kernel/mt76/patches/0001-mac80211-mask-nested-A-MSDU-support-for-mesh.patch

This will place the patch shared above under package/kernel/mt76/patches/. You can then run make to build the firmware again.

Thank you for the response man but noob question, do you need to create this directory first /path/to/openwrt/source/ , right?

I did create the directory manually, cd into it and then invoke the wget command but I received, "No such file or directory" . Sorry man, I'm still learning though I already have an idea by following this guide: https://openwrt.org/docs/guide-developer/toolchain/use-patches-with-buildsystem but seems like I failed or need to understand it deeper.

P.S. on the tutorial, I managed to edit the mac80211.c then quilt refresh but I'm stuck on this command: "make package/kernel/mac80211/{clean,compile} package/index V=s" it resulted on compile error when I invoke this.