DietPi
DietPi copied to clipboard
MichaIng
Erratic Ethernet Performance on NanoPi Fire3
Hello,
I am adding 20 new DietPi installs to an existing cluster.
I configured up two of the new nodes, and upon first boot had incredibly high/erratic ping times on the newest image; identical on both devices. Kernel version on the non-working install is 4.14.141-s5p6818 on DietPi v6.31.2.
As a related note, the other 20 NanoPi Fire 3's in my cluster are all working find on the same switch running Dietipi v6.15 on kernel 4.14.70-s5p6818 and have had nearly 100+ days of uptime with no network dropouts.
Pinging Gateway:
root@DietPi-NewCluster-Test:~# ping 172.16.69.1 PING 172.16.69.1 (172.16.69.1) 56(84) bytes of data. 64 bytes from 172.16.69.1: icmp_seq=1 ttl=64 time=280 ms 64 bytes from 172.16.69.1: icmp_seq=2 ttl=64 time=544 ms 64 bytes from 172.16.69.1: icmp_seq=3 ttl=64 time=554 ms 64 bytes from 172.16.69.1: icmp_seq=4 ttl=64 time=6118 ms 64 bytes from 172.16.69.1: icmp_seq=5 ttl=64 time=5106 ms 64 bytes from 172.16.69.1: icmp_seq=6 ttl=64 time=4082 ms 64 bytes from 172.16.69.1: icmp_seq=7 ttl=64 time=3058 ms 64 bytes from 172.16.69.1: icmp_seq=8 ttl=64 time=2034 ms 64 bytes from 172.16.69.1: icmp_seq=9 ttl=64 time=1010 ms
Device: NanoPi Fire3-LTS (Friendlyelec)
Diet Pi Version: DietPi v6.31.2 : 16:27 - Sat 07/11/20
uname -a:
Linux DietPi-NewCluster-Test 4.14.141-s5p6818 #7 SMP Mon Sep 2 07:10:36 CEST 2019 aarch64 GNU/Linux
Any assistance or ideas would be welcome! Can't seem to locate the image I used to burn v6.15 either :)
Hi,
many thanks for your report. Just for clarification and to avoid a misunderstanding. The version of DietPi is not directly related to the kernel version. DietPi did not and will not create own kernels. The kernel is provided by the underlying Debian base image, always.
Joulinar,
Thanks for the clarification. Upon further investigation, it appears that Armbian has this as a known issue for the board.
https://forum.armbian.com/topic/13624-nanopi-fire-3-unstable-gigabit-ethernet
What I can say is that it works on Dietpi v6.15 :)
Thanks for any assistance!
I'm not sure how we can mitigate this from DietPi side if it's a bug inside Armbian kernel
@Joulinar Thanks for the confirmation -- any idea where I might be able to find a v6.15 image that works with the built in ethernet or tips on where should start looking to build an older image?
I think this may be a breaking issue that should probably be listed on the install page, so folks don't come across it unexpectedly.
Did you try to upgrade to latest kernel version? This should be 4.14.180:
apt update
apt full-upgrade
The legacy image is here: REMOVED OBSOLETE LINK Debian Stretch, Linux 4.4.49, unmaintained kernel, but this should be the image that you run with DietPi v6.15 as the new Armbian-based image was created later, AFAIK.
... ahh I just see that it has wrong kernel packages installed, because Armbian switched their package naming back and forth. To upgrade to latest 4.14.180, do the following:
apt update
apt install linux-buster-root-legacy-nanopifire3 linux-u-boot-nanopifire3-legacy linux-dtb-legacy-s5p6818 linux-image-legacy-s5p6818
apt purge linux-buster-root-next-nanopifire3
apt purge linux-u-boot-nanopifire3-next
apt purge linux-dtb-next-s5p6818
apt purge linux-image-next-s5p6818
A bid strange that the new variant is called "legacy" while the old one is called "next". In the past they named the old non-mainline packages without prefix/suffix and when starting with the mainline kernel packages, they named them "next". Now they name any pre-5.X kernel "legacy" (even that it is the most current and maintained one) and the 5.X packages are named "current", which are not available yet for NanoPi Fire3. The "next" packages are not updated anymore.
Means I need to rebuild the image to contain the correct maintained package.
New image is up for testing: https://dietpi.com/downloads/images/testing/DietPi_NanoPiFire3-ARMv8-Bullseye.7z @cmoski probably you find time to test if it brings any benefit to Ethernet connections.
@MichaIng First let me say -- thank you so much for your work on DietPi. Really appreciate everything that you folks do for the community.
Testing this image now, I'll load it up on two of the devices and let you know the results.
Thanks again! cmoski
Looks like this fixed things for me; at least. Everything seems to be responding normally! Excited to be able to run the latest docker version :-)
Very nice to hear and many thanks for testing. Let me know if you run into any issues with this image, so I can evaluate whether to move to stable place quickly.
@MichaIng: Could you define a couple of tests resp. measurements we should do? Be open with your wishlist. :-) Then we could divide these tasks between testers to give feedback.
I think it is not an issue anymore with the "legacy" kernel package set, which ships the latest Linux versions. Not sure why I marked it for testing 😉.
@MichaIng This issue is up again with the latest Diet-pi image (Bullseye) for the NanoPi Fire3. I can SSH into the nano, but it responds very slowly.
In contrast, when I run the Friendly Elec image 64bit (Ubuntu 20.04) it runs great.
Is there anything you can do? I really enjoy diet-pi on my other SBCs, and would like to have a recent version running on my fire3 as well.
Can your script install diet-pi on an existing Ubuntu install or only on Debian ?
BTW, I'm not able to download any previous versions of the image from your site, as the links in this thread no longer exist.
Thanks.
Can your script install diet-pi on an existing Ubuntu install or only on Debian ?
It supports only Debian.
BTW, I'm not able to download any previous versions of the image from your site, as the links in this thread no longer exist.
We provide only downloads for the latest images.
I can SSH into the nano, but it responds very slowly.
I wanted to have a look into recent kernel packages and Armbian forum, but all Armbian websites are currently down. Can you show:
apt show linux-dtb-current-s5p6818 linux-image-current-s5p6818 linux-u-boot-nanopifire3-current
Can your script install diet-pi on an existing Ubuntu install or only on Debian ?
It supports only Debian.
BTW, I'm not able to download any previous versions of the image from your site, as the links in this thread no longer exist.
We provide only downloads for the latest images.
I can SSH into the nano, but it responds very slowly.
I wanted to have a look into recent kernel packages and Armbian forum, but all Armbian websites are currently down. Can you show:
apt show linux-dtb-current-s5p6818 linux-image-current-s5p6818 linux-u-boot-nanopifire3-current
This is what I got on a fresh install:
root@DietPi:~# apt show linux-dtb-current-s5p6818 linux-image-current-s5p6818 linux-u-boot-nanopifire3-current
N: Unable to locate package linux-dtb-current-s5p6818
N: Unable to locate package linux-image-current-s5p6818
N: Unable to locate package linux-u-boot-nanopifire3-current
N: Unable to locate package linux-dtb-current-s5p6818
N: Unable to locate package linux-image-current-s5p6818
N: Unable to locate package linux-u-boot-nanopifire3-current
E: No packages found
root@DietPi:~#
Ah, now Armbian's website is up again. Indeed there is only one "legacy" kernel package for this SoC. The U-Boot package is not available anymore at all, as also the SBC is not in the list of officially supported by Armbian (i.e. no maintainer). The last image as well as the last kernel packages are more than one year old. So whatever you are facing, it was there for a long time.
Found the related issue at Armbian forum: https://forum.armbian.com/topic/13624-nanopi-fire-3-unstable-gigabit-ethernet/ In short: No maintainer, no support from FriendlyELEC, many issues and zero development with both kernels, Armbian and stock from FriendlyARM. It doesn't look well for this neat little SBC 😢. Stock images are Linux 4.4 btw, which alone makes it impossible as basis.
Can you check:
ethtool -k eth0 | grep tcp
Yeah, the kernel on the FriendlyElec is old (4.4), but its Ubuntu 20.04. I can live with that... But I like Dietpi's features, so it would be cool to have it running. I know that this is an old problem, but I recently made use of two of my Fire3's, that were collecting dust (maybe because of this issue, I can't recall).
Anyways, this is what ethtool brought up:
cp-segmentation-offload: off
tx-tcp-segmentation: off [fixed]
tx-tcp-ecn-segmentation: off [fixed]
tx-tcp-mangleid-segmentation: off [fixed]
tx-tcp6-segmentation: off [fixed]
The kernel on the working FriendlyElec image is: 4.4.172-s5p6818 while on the the Dietpi is: 4.14.180-s5p6818. Could it be that the older kernel doesn't have this problem?
Is there a way to install the older kernel?
Seems to be indeed off already, as expected. Just to rule it out, could you try:
ethtool -K eth0 rx off tx off
See whether this makes a difference. And:
ethtool -K eth0 rx on tx on
whether this does.
Unfortunately not much difference... The network is still pretty choppy, with both settings.
Could it be that the older kernel doesn't have this problem?
It is not so much older vs newer, but stock vs Armbian. The stock kernel of course works, but has increasing other issues with recent distro versions. The newer kernel has less of such issues but obviously others which were never resolved due to lack of development time and support.
Does it help to pin Ethernet speed to 100 Mbits?
ethtool -s eth0 speed 100 duplex full autoneg off
I just read again carefully through the thread. Strange that it worked well in the past with this very same kernel, at least for @cmoski 🤔. Probably there are other factors relevant, like power supply or other attached devices?
100 Mbits seems to help! Network is more stable and consistent, but slower than the FriendlyElec image... I've got a Gigabit Internet connection, so I feel the difference (for example when installing packages).
I have ruled out other factors: I have two SD cards, one with the FriendlyElec image, and another with Dietpi on it. When I use the friendlyelec, all works well. When I use the Dietpi, its all slow.
100 Mbits seems to help!
Ah nice. I mean sad to have 100 Mbit only then, but better than several seconds reaction time.
When I use the friendlyelec, all works well. When I use the Dietpi, its all slow.
It's clear that it is not an issue with FriendlyELEC kernel, but the question is why it is not an issue on all DietPi/Armbian systems. With the currently used kernel you are the first one reporting it, and the (hidden) Armbian NanoPi Fire3 download page mentions it as "occasional" issue. If it is present in all cases, we could add the 100 Mbit limit to our image, but limiting Ethernet speed on systems where the issue is not present isn't good. So we need to find out which other factors trigger the issue with Armbian's kernel.
The SD card can be nearly ruled our to have any impact, I'm more thinking about the power supply, attached USB devices, HDMI devices or something on the GPIO pins. Do you have any such attached and do you have another PSU or even just another higher quality USB cable to try with? Also can you please post the kernel log:
dmesg
Probably it gives a hint, especially if voltage is insufficient at any stage.
I have nothing connected to the Fire3, other than micro USB for power and an ethernet cable (cat6 connected to my 1gig switch).
I'm running SSH from a computer also connected to the same switch. Again, if I plug in the SD card that has the FreindlyELEC image on it, it works well. When I switch just the SD card (both of same manufacturer, type and size) with the Dietpi on it, it works slow.
As to your question, maybe it's a Dtb issue for the Fire3?
When running ethanol on the friendlyelec image, it gives this:
Supported ports: [ TP MII ]
Supported link modes: 10baseT/Half 10baseT/Full
100baseT/Half 100baseT/Full
1000baseT/Half 1000baseT/Full
Supported pause frame use: No
Supports auto-negotiation: Yes
Supported FEC modes: Not reported
Advertised link modes: 10baseT/Half 10baseT/Full
100baseT/Half 100baseT/Full
1000baseT/Half 1000baseT/Full
Advertised pause frame use: No
Advertised auto-negotiation: Yes
Advertised FEC modes: Not reported
Link partner advertised link modes: 10baseT/Half 10bas
eT/Full
100baseT/Half 100b
aseT/Full
1000baseT/Full
Link partner advertised pause frame use: Symmetric
Link partner advertised auto-negotiation: Yes
Link partner advertised FEC modes: Not reported
Speed: 1000Mb/s
Duplex: Full
Port: MII
PHYAD: 7
Transceiver: external
Auto-negotiation: on
Supports Wake-on: g
Wake-on: d
Current message level: 0x0000003f (63)
drv probe link timer ifdown ifup
Link detected: yes
But with the dietpi image (currently on 100Mbits) it gives the following:
Settings for eth0:
Supported ports: [ TP MII ]
Supported link modes: 10baseT/Half 10baseT/Full
100baseT/Half 100baseT/Full
1000baseT/Half 1000baseT/Full
Supported pause frame use: Symmetric Receive-only
Supports auto-negotiation: Yes
Supported FEC modes: Not reported
Advertised link modes: 100baseT/Full
Advertised pause frame use: No
Advertised auto-negotiation: No
Advertised FEC modes: Not reported
Speed: 100Mb/s
Duplex: Full
Port: MII
PHYAD: 7
Transceiver: external
Auto-negotiation: off
Supports Wake-on: ug
Wake-on: d
Current message level: 0x0000003f (63)
drv probe link timer ifdown ifup
Link detected: yes
The ethtool outputs are identical when skipping those parts affected by forcing 100 Mbit. If you have no other devices attached, then PSU or PSU cable and dmesg output are the last ideas I have for now to test/debug the issue.