Can't ssh to precompiled k4.19.99. But ssh in Jessie image works ok

[logon84]

Hi. I have installed the Jessie image on my WD Mybooklive single disk (2TB). I connected thru ssh, configured all the stuff and it is working very nice. The problem I am facing is that when replacing /boot/apollo3g.dtb and uImage on /boot folder with the ones in the precompiled 4.19.99 kernel, then I can ping the device but ssh refuses connection. Any idea what is happening?

[jlores]

Hello,

I think that you forget change swap. Please confirm that your swap is /dev/sda3, or chage the command.

"Please note that the swap space must match the kernel block size. So, if new kernel has a different page size than the previous one, you need to re-initialize swap space. Assuming the standard MBL disk layout, swap space is on /dev/sda3. The mkswap command will read the kernel page size, so no need to pass the --pagesize option if you already booted from the new kernel. Since 4.9.x and 4.19.x have different default page sizes in the default kernel configuration files (for now), 64K and 16K respectively, this issue might arise as you swap kernels.

mkswap --pagesize 65536 /dev/sda3"

Best regards

[logon84]

Hello,

I think that you forget change swap. Please confirm that your swap is /dev/sda3, or chage the command.

"Please note that the swap space must match the kernel block size. So, if new kernel has a different page size than the previous one, you need to re-initialize swap space. Assuming the standard MBL disk layout, swap space is on /dev/sda3. The mkswap command will read the kernel page size, so no need to pass the --pagesize option if you already booted from the new kernel. Since 4.9.x and 4.19.x have different default page sizes in the default kernel configuration files (for now), 64K and 16K respectively, this issue might arise as you swap kernels.

mkswap --pagesize 65536 /dev/sda3"

Best regards

should be pagesize 16384 for 4.19 right?

EDIT: Just tried re-initializing swap with both 65536 and 16384 pagesizes, but still ssh refuses to connect to mbl. I also confirm my swap partition is in /dev/sda3

[jlores]

Yes, 16384 for 4.19.

Now I testing one 5.10.x kernel.

Best regards

[jlores]

Sorry and don't read your edit.

Please, try to comment "#" your swap on fstab,

[ewaldc]

Hmm, I am aware of ssh issues when you just deploy the Debian image and do the typical things like change the hostname. Hence in the installation guidance we recommend to perform the commands below.

# clean up ssh
rm /etc/ssh/ssh_host* _
dpkg-reconfigure openssh-server
systemctl restart ssh 

Do these commands help to restore ssh? Does ssh -v reveal anything useful ?

It's possible that a kernel change could break ssh, (improper swap space block size, the cipher being used no longer in new kernel, missing kernel module) but if you have not changed anything to the original image and have the proper swap space block size, I'm clueless... I have switched between 4.9.x and 4.19.x hundreds of times (having mkswap /dev/sda3 in the startup files).

[ewaldc]

@jlores, would be interested to hear about your experience with 5.10. Before my MBL died, I tried desperately to match 4.X performance level on 5.4. It took a lot of patches (TCP, ATA, block) to get within 10% of 4.19.x . I was able to fix 16K pages but 64K pages was broken beyond repair. I also could not get kernel profiling working... Hopefully you are more successful !! Ewald

[logon84]

Hmm, I am aware of ssh issues when you just deploy the Debian image and do the typical things like change the hostname. Hence in the installation guidance we recommend to perform the commands below.

# clean up ssh
rm /etc/ssh/ssh_host* _
dpkg-reconfigure openssh-server
systemctl restart ssh 

Do these commands help to restore ssh? Does ssh -v reveal anything useful ?

It's possible that a kernel change could break ssh, (improper swap space block size, the cipher being used no longer in new kernel, missing kernel module) but if you have not changed anything to the original image and have the proper swap space block size, I'm clueless... I have switched between 4.9.x and 4.19.x hundreds of times (having mkswap /dev/sda3 in the startup files).

Hi. The thing is, ssh works perfect in k4.9. After I replace uImage and apollo file with the ones of the k4.19 build, then I cannot ssh, only ping. If I switch back to k4.9 again, just overwriting uImage and apollo files, ssh continues to work as it was.

@jlores Also tried commenting the swap partition in fstab, ssh still refuses to accept connections.

my actual fstab:

## increasing to 100M maximum 
tmpfs	/tmp tmpfs rw,size=100M 0 0
#tmpfs	/var/log tmpfs nodev,nosuid,size=20M 0 0
# /var/cache/samba --> /run/cache/samba
#tmpfs	/var/cache/samba tmpfs defaults,noexec,nosuid,nodev,mode=0755,size=100M 0 0

/dev/sda1	/	ext3	defaults,noatime,nodiratime,barrier=1	0 0
#/dev/sda3	swap	swap	defaults	0 0
/dev/sda4	/DataVolume	ext4	exec,rw,noatime,nodiratime,data=writeback,barrier=0,errors=remount-ro,suid	0 1
/DataVolume/cache	/CacheVolume 	none defaults,bind 0 0
/DataVolume/shares	/shares 	none defaults,bind 0 0
/DataVolume/shares	/nfs 		none defaults,bind 0 0

[ewaldc]

Next to replacing uImage/apollo.dtb, you do need to run either "mkswap /dev/sda3" or "swapon --fixpgsz /dev/sda3" if the page sizes are different (e.g. from /etc/rc.local) at boot time. Just commenting it out in /etc/fstab might not be enough.

What does "ssh -v" say? Is /var/run/sshd created after boot with the new kernel? Could be some defect (e.g. link). Does /var/run point to /run (/run should not be a tmpfs) ?

I do recall having seen the issue once before, but I can't remember what caused it...

[logon84]

Next to replacing uImage/apollo.dtb, you do need to run either "mkswap /dev/sda3" or "swapon --fixpgsz /dev/sda3" if the page sizes are different (e.g. from /etc/rc.local) at boot time. Just commenting it out in /etc/fstab might not be enough.

What does "ssh -v" say? Is /var/run/sshd created after boot with the new kernel? Could be some defect (e.g. link). Does /var/run point to /run (/run should not be a tmpfs) ?

I do recall having seen the issue once before, but I can't remember what caused it...

Tried adding swapon --fixpgsz /dev/sda3 to rc.local, still no success. The problem to check ssh -v and the rest of folders you say is.....if I can't ssh to the device then how can I enter those commands?

EDIT: and interesting test I just did.....it's not just an ssh issue, I can't access the samba shares with 4.19.99 kernel as I can with 4.9. So it is not just an ssh issue, but the device seems that is not booting properly with that kernel.

[ewaldc]

If you have a Linux OS, you can use "ssh -v" to test login to the MBL. The way you can add a command to e.g. /etc/rc.local is to take the disk out and mount it on a Linux machine. Alternatively, if you implemented the netconsole, you can select which kernel to boot. Even better is to boot from TFTP e.g. from your wireless router. But, the more logical explanation is that the box does not boot properly at all, rather than just a ssh issue. Things to check:

• Do you have a MBL Duo? If so you need the dtb for duo.
• As we said before, it can not boot the new kernel if you don't add "mkswap /dev/sda3" to your startup files (e.g. /etc/rc.local)
• Have you tried with 4.19.96 and 4.9.119_hdd_led ? The last one is probably one of the best kernels.

Ewald

[logon84]

If you have a Linux OS, you can use "ssh -v" to test login to the MBL. The way you can add a command to e.g. /etc/rc.local is to take the disk out and mount it on a Linux machine. Alternatively, if you implemented the netconsole, you can select which kernel to boot. Even better is to boot from TFTP e.g. from your wireless router. But, the more logical explanation is that the box does not boot properly at all, rather than just a ssh issue. Things to check:

* Do you have a MBL Duo?  If so you need the dtb for duo.

* As we said before, it can not boot the new kernel if you don't add "mkswap /dev/sda3" to your startup files (e.g. /etc/rc.local)

* Have you tried with 4.19.96 and 4.9.119_hdd_led ?  The last one is probably one of the best kernels.

Ewald

Hi. My device is single bay 2TB MBL. I am using apollo3g.dtb file with crc32 0x7fbb0cf1 when trying to boot k4.19.99. Using apollo3dtb file with crc32 0xadcd08c0 when succesfully bootting to k4.9.99 Tried also swapping from swapon command to mkswap /dev/sda3 in rc.local, no luck. Tried booting 4.9.119_hdd_led IT WORKS Tried booting 4.19.96, no luck. SSH -v gives:

debug1: Connecting to 192.168.1.3 [192.168.1.3] port 22.
debug1: connect to address 192.168.1.3 port 22: Connection refused
ssh: connect to host 192.168.1.3 port 22: Connection refused

I will probably try to solder a header to the serial port soon to see if it comes with something interesting.

[ewaldc]

"connection refused" may be misleading. U-boot brings up the LAN with an IP address, so ssh does see an active host, but there is no ssh daemon (sshd) to take the connection, so it's totally possible it does not boot. If you don't want to solder, read up the section on "netconsole".

[logon84]

Ok, I did a new boot.scr with nc support:

setenv nc 'setenv bootdelay 10; setenv stderr nc; setenv stdout nc; setenv stdin nc'
setenv ipaddr '192.168.1.3'
setenv ncIPWLan '192.168.1.20'
setenv ncMacWLan '60:57:18:FD:30:D1'
setenv ncIPLan '192.168.1.2'
setenv ncMacLan '<<my mac>>'

if itest -z "${boot_count}"; then setenv boot_count 0; fi  
if itest ${boot_count} == 0; then setenv boot_count 1; else setenv boot_count 2; fi
saveenv
run nc
setenv bootargs 'root=/dev/sda1 earlycon earlyprintk rw rootfstype=ext3 rootflags=data=ordered ipv6.disable=1'
setenv debugargs 'setenv bootargs debug rootdelay=5 panic=10 debug ignore_loglevel log_buf_len=1M ${bootargs}'
setenv bootargs_lan 'setenv bootargs netconsole=6663@${ipaddr}/,6664@${ncIPLan}/${ncMacLan} ${bootargs}'
setenv bootargs_wlan 'setenv bootargs netconsole=6663@${ipaddr}/,6664@${ncIPWLan}/${ncMacWLan} ${bootargs}'
setenv load_sata 'sata init; ext2load sata 1:1 ${kernel_addr_r} /boot/uImage; ext2load sata 1:1 ${fdt_addr_r} /boot/apollo3g.dtb'
setenv load_sata_rcvr 'sata init; ext2load sata 1:1 ${kernel_addr_r} /boot/uImage.safe; ext2load sata 1:1 ${fdt_addr_r} /boot/apollo3g.safe.dtb'
setenv load_tftp 'tftp ${kernel_addr_r} ${bootfile}; tftp ${fdt_addr_r} ${fdt_file}'
setenv boot_kernel 'run bootargs_lan addtty; bootm ${kernel_addr_r} - ${fdt_addr_r}'

if itest ${boot_count} == 1; then echo "=== Loading Default Kernel ==="; run load_sata; else echo "=== Loading Recovery Kernel ==="; run load_sata_rcvr; fi
printenv
run boot_kernel

Being apollo3g.safe.dtb and uImage.safe the ones from the working 4.9.119 kernel, and uImage and apollo3g.dtb the ones from 4.19.99 kernel. On the remote computer I only see the booting log from Uimage.safe, not a single char from the first try with 4..19.99. This confirms to me that kernel 4.19 is not loading at all.

[logon84]

Could it be a wrong apollo3g.dtb in to the kernel compressed files? I mean, for what device is the dtb present in the precompiled kernels, single or duo?

[ewaldc]

Nice work! This explains the ssh mystery at least. Is /dev/sda1 formatted in ext3? It should be cause it's set on your boot args.... but maybe I have removed ext2 from the 4.19 kernels.

[logon84]

Yes, sda1 is ext3 fs. Can you confirm dtb files attached with precompiled kernels is for single hd mbl?

[ewaldc]

The one in the tar file seems OK and also consistent between 4.19.96 and 99. It is different from 4.9. Name: apollo3g.dtb Size: 11515 bytes (11 KiB) SHA256: 3CD14368B2C86CF5BC323D3341D0A097EE488692F605B367852A9AD54458FBCA

[ewaldc]

Unfortunately my MBL died, about a year ago so I can no longer test., but I just acquired a second hand one and will test as soon as it arrives.