ZenWiFi XT8 v1 only 3 of 4 CPU cores in use [Updated]

[robbbaxley]

Router Model Affected Models: ZenWiFi XT8 v1

Firmware Version Affected RT-AX95Q_3004_388.4_0-gnuton1_puresqubi

Is this bug present in upstream Merlin releases too? No

Describe the bug A few days ago, I reported that only 3 of 4 CPU cores are being utilized in this firmware version. All cores are being utilized in all releases of the official Asus firmware. I was doing some additional analysis and testing and found that when I transferred data to a USB drive (SSD) connected to the USB port on the router, 3 of the 4 cores would operate at or near 100% with the 4th core finally showing some utilization but only 3% to 5%. Transfer speed was slow at an average of 33Mb/s.

To Reproduce Steps to reproduce the behavior:

1. Go to 'Network Map'
2. Click on 'Status' tab
3. Watch CPU utilization graph
4. See error

Expected behavior A 4 CPU cores to be consistently utilized and better transfer speed to USB drive.

Screenshots Attached

[dacootmeister]

It's long known issue, no fix until now The 4th core is quieter then the first 3, but low level it still does something from linux htop:

[robbbaxley]

It's odd, though, that all 4 cores are move evenly utilized on the standard Asus WRT firmware and I never saw utilization spike to (and stay at or near 100% utilization. To me, it appears that threading/scheduling is somehow different.

[jazha]

While transferring files internally on my network, from a wired source to a wifi destination, I get life from core 4 in gui.

[robbbaxley]

This is so odd!

Can I ask which version of firmware you are using? Do you know which version of hardware you have? Mine is v1 and I see utilization similar to what you are showing, but only when using the standard firmware. I have verified standard core utilization on Asus 3.0.0.4.388.23285 firmware.

[jazha]

Im on a V1 aswell.

I run 3004.388.4_0-gnuton1

Not sure how to extract the exact numbers, but its the only one gnuton released for 3004 series of FW I believe

[robbbaxley]

I took this capture just now. The only time I see any utilization at all on Core-4 is when I am transferring data to the USB disk over the network. The first 3 cores work at nearly 100% during the transfer but Core-4 is barely utilized. I have 5 devices, all AP's and all connected over Wired backhaul (1 device is currently offline due to a backhaul failure). I have 3 XT8's and 2 XT9's. All version 1's. The XT9's are using the default Asus firmware.

[jazha]

Could be core is locked to specific tasks, like running ethernet/wifi connections etc. I dont know enough to figure that out. I wouldnt worry about low load unless you have issues with the router not working properly. Someone more familiar with the workings could probably shed some light on the 4th cores tasks?

[robbbaxley]

Here is another test I completed just now. First screen capture is after plugging in a USB drive. Nothing else. Notice how CPU utilization increased though I don't know why. In the second screen capture, it shows data transfer over the wired network to the USB drive. CPU utilization nearly maxes out the first 3 cores. The 4th core finally shows some utilization but only just.

[robbbaxley]

I should add, my only concern is that the data transfer rate over the wired network to the USB drive is much slower than expected.

[jazha]

What type of drive do you have connected? Maker/model/connection/portable/powered/size/file system etc?

Could be something as simple as different drives just work differently unfortunately.

[robbbaxley]

In the very first screen capture, for that test I used a 7.68TB Micron 5300 Pro SSD (550MB/s write speed). In the most recent test, I used a PNY 512GB Elite-X Fit USB 3.1 Flash Drive (200MB/s write speed).

Both drives had a new partition created and formatted NTFS in Windows 11 Pro. I did attempt to use the PNY drive as exFAT but the Asus router wouldn't use it with that file system.

[jazha]

For sure none of those should cause a slowdown in transfer speed. I have really no clue what throughoutput the router should in any given case. I feel you should get what the ethernet port can give, which should be just above 100MB/s transfer in "perfect" conditions. Personally I use a storage drive connected to a computer internally on the network, and not directly in the router, so I dont have any of the issues you mention.

[robbbaxley]

Yeah, I am stumped too. I tested my network card speed and it maxes out at just over 2.1Gb/s so I am confident that's not the bottleneck. I am pretty sure that the issue lies somewhere in the firmware. Since the 3 working cores max out I am reasonably confident that there is an issue with the USB and/or CPU threadding.

[robbbaxley]

Jazha, do you have a USB drive you can plug into the XT8 and see if you get the same results I am seeing?

[jazha]

Yeah, I am stumped too. I tested my network card speed and it maxes out at just over 2.1Gb/s so I am confident that's not the bottleneck. I am pretty sure that the issue lies somewhere in the firmware. Since the 3 working cores max out I am reasonably confident that there is an issue with the USB and/or CPU threadding.

Or worse, the HW is not good enough to support such speeds, resulting in offload to the cpu for things that should go via the storage controller (if I understand correctly good HW storage controllers should be capable of handling this easy, but maybe they put a cheap one in or did some other trick to keep costs down)

Jazha, do you have a USB drive you can plug into the XT8 and see if you get the same results I am seeing?

I would do this in a heartbeat if I wasnt running AMTM on it and I rely on the single port to hold an USB stick for the storage requirements for Diversion , Skynet etc.

[soelu]

I am running the stock firmware still and see the same effect. So i assume this is not a Merlin but a Asus Stock Firmware "Issue"

[Dodgydrains]

Seems fine to me on stock 23285 FW. This is just running a speedtest on speedtest.net.

[robbbaxley]

@soelu, are you on hardware v1 or v2? I have v1 and, like @Dodgydrains, the cores operate properly on stock firmware for me.

[Dodgydrains]

@robbbaxley I am also on V1 hardware and stock FW and cores seem fine on it.

[Dodgydrains]

This seems to be working for me on latest 3004.388.5_0-gnuton0_beta1, I did dirty flash from stock 23285 FW.

[soelu]

I have Hardware version V1. It uses the 4th CPU, but unproportionally. This is during a Zoom Call and running a speedtest in parallel.

[gnuton]

The load is not spread across all cpus, since cpu affinity kicks in to keep some processes in some threads. This has been manually fine tuned by ASUS and it's what you see in all ASUS routers.

This has been already discussed a while ago and RMerl has replied to this too with some tests. https://www.snbforums.com/threads/rt-ac66u_b1-not-using-2nd-cpu-core.39670/

[soelu]

So I guess this is a non-issue for the Merlin Firmware then?

[gnuton]

@soelu according to what I have read in the post there were not advantages. if you have specific applications which you wnna fine tune for using more threads you may try taskset

taskset -p 1 "$pid";

You can get the pid of your app running "ps" from the command line and you can replace 1 with the bitmask associated to the cores you want to use. You can also install irqbalance.

[robbbaxley]

I can also confirm that, as @Dodgydrains noted, the ZenWifi XT8 Beta firmware version RT-AX95Q_3004_388.5_0-gnuton0_beta1_puresqubi, is working much better as it pertains to thruput to the USB attached storage.

I would like to note the UL and DL appear to be backwards to me (I am uploading files to the USB drive from a PC). Any thoughts?

[Dodgydrains]

I can also confirm that, as @Dodgydrains noted, the ZenWifi XT8 Beta firmware version RT-AX95Q_3004_388.5_0-gnuton0_beta1_puresqubi, is working much better as it pertains to thruput to the USB attached storage.

I would like to note the UL and DL appear to be backwards to me (I am uploading files to the USB drive from a PC). Any thoughts?

Mine D/L and U/L seems right way round when doing a speedtest, but I did dirty flash over official FW 23285

[robbbaxley]

@Dodgydrains, I just checked and also verify that the Ul/DL looks correct when in internet download/streaming/speed-test. So the graph is probably correct and it just looks backwards to me.

Now if we could just figure out WHY on the Merlin firmware 3 cores are utilized heavily when, in the Asus released firmware, the load is much more evenly balanced across all 4 cores. At least USB transfers are working quite a bit better with this Beta firmware.

[robbbaxley]

I wanted to post an update regarding the new firmware version RT-AX95Q_3004_388.5_0-gnuton1_puresqubi.w.

Still showing 3 of 4 cores with high CPU utilization during file transfers to a USB drive connected to the primary node. The 4th core shows o% to 4% utilization during data transfer to USB storage. This is the same as with the previous firmware version.

I set the USB version to both USB 2.0 and USB 3.0 in the primary node (where the drive was connected) and tested with no change in transfer speed. All nodes connected via hardwire at 2.5Gb/s. Computers used to test was 1 Windows 11 desktop connected via 1Gb/s hardwire and 1 Linux laptop connected via wireless. There was no difference in CPU utilization on the primary node with the connected storage drive and no difference in data transfer rates.

[LarryRosen108]

I have seen similar results with higher node 4 % but without any real impacts to usage of my 2 node XT8 environment. My transfer was done using winscp from a laptop with 1gb Ethernet connection directly to my main node that contains the USB 3.0 stick. This is on the latest .5 release.

---

Sent from my mobile device so please pardon autocorrect and brevity

On Thu, Jan 4, 2024 at 12:22 PM robbbaxley @.***> wrote:

I wanted to post an update regarding the new firmware version RT-AX95Q_3004_388.5_0-gnuton1_puresqubi.w.

Still showing 3 of 4 cores with high CPU utilization during file transfers to a USB drive connected to the primary node. The 4th core shows o% to 4% utilization during data transfer to USB storage. This is the same as with the previous firmware version.

I set the USB version to both USB 2.0 and USB 3.0 in the primary node (where the drive was connected) and tested with no change in transfer speed. All nodes connected via hardwire at 2.5Gb/s. Computers used to test was 1 Windows 11 desktop connected via 1Gb/s hardwire and 1 Linux laptop connected via wireless. There was no difference in CPU utilization on the primary node with the connected storage drive and no difference in data transfer rates.

— Reply to this email directly, view it on GitHub https://github.com/gnuton/asuswrt-merlin.ng/issues/500#issuecomment-1877484335, or unsubscribe https://github.com/notifications/unsubscribe-auth/A2XR7N2TEWUW2M5STZPD2L3YM3QL7AVCNFSM6AAAAAA72QNZ56VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQNZXGQ4DIMZTGU . You are receiving this because you are subscribed to this thread.Message ID: @.***>

[swwwong]

@robbbaxley

you can manaul adjust some task to the one with low usage .