clients not booting after recent Raspbian Stretch upgrade

[gbrickell]

Hi - all of our clients have stopped booting after a recent update/upgrade to our customised Raspbian Stretch OS. They all hang after the 4 Raspberries are shown on the client screen and then they 'hang' with some sort of activities that seem to cycle forever but the boot does not complete. This happens with both oldish RPi3B's and newer 3B+'s.

Fortunately I have been taking occasional .tar.xz snapshots so I was able to bring back a version from the 10th July into PiServer and assign this to one of the B+ clients - which booted perfectly with this older version - but just to cross check I then re-ran a full update/upgrade cycle on the OS and then this updated 10th July version also failed to boot.

Then to try and see whether it was some peculiarity with our modded Stretch version I then installed fresh copies of both 'Raspbian Lite' and 'Raspbian Full' (labelled as 2019-04-08) and unchanged the 3B+ client booted OK from both these OS - but then I did a full update/upgrade to both of these in turn and, yes, then these also failed to boot on a client machine.

I don't think I can diagnose this much more but it seems to be a pretty consistent and repeatable issue with an upgrade that happened after the 10th July - and hopefully you will be able to reproduce the problem for yourself. Our PiServer use is for a School club and obviously School is out for a few weeks so no immediate urgency.

Thanks

geoff

[maxnet]

As a workaround:

In the Piserver GUI go to "tab software" -> select Raspbian -> "shell"

leafpad /boot/cmdline.txt

Add to the same line as other options: modprobe.blacklist=bcm2835_v4l2

Is caused by that you are received a 4.19 kernel when you do apt-get upgrade, and there is something with the bcm2835_v4l2 module that comes with that (or the firmware functions it calls). Existing bug report: https://github.com/raspberrypi/linux/issues/3067

[gbrickell]

Thanks! That has sorted the problem .. but will this restrict any kind of camera or other usage or does an earlier version still get loaded? and presumably a later version once a full fix is implemented.

[maxnet]

Will disable any camera usage you do through v4l2. Not sure about other ways to access camera (openmax et al)

[gbrickell]

hmmm OK - we have a number of Image Taking exercises that the students can do with a cheap low res USB camera and the example python code for simple video clip taking, currently uses avconv and the v4l2 format i.e. the typical command string is: 'avconv -f video4linux2 -s 640x480 ....etc'

But I've just checked and this is still working OK with the now booting upgraded OS and the video streaming exercises we do use Flask and openCV and these are also still working - so all good so far :-)

[maxnet]

An USB webcam should indeed be fine. Problem is limited to accessing the official Raspberry Pi camera connected by CSI ribbon cable through v4l2.

[gbrickell]

OK understood and thanks again - should I assume that this issue should remain open until a full fix is in place?

[maxnet]

Yes. Although it's technically a duplicate of the issue in the Linux repository, I think it is fine to leave it open for now, so that others know. Since it mainly affect Piserver users, they are more likely to look for existing issues here than there.

[trijenhout]

i dont have a pi server, i have a opensuse15.1. Aldo i have 3 pi's running true pxe/nfsroot boot for a year now... . i do have the same problem. i runinng the 2019-04-08-raspbian-stretch.img on one whit no updates for a kodi to home theater system... the other 2 running 2019-07-10-raspbian-buster-full.img,2019-07-10-raspbian-buster-lite.img. one as a test / minimal desktop the other whit munin-client mainly for ups + running yacy for fun. cmdline.txt option: modprobe.blacklist=bcm2835_v4l2. the desktop whit config.txt:gpu_mem=256 the other whit mem=16 those 2 are up to date weekly manual... whit dose options i got them quit stable.. not optimal....

[gorky]

Similar setup as trijenhout -- although running one PI with an HD as a TFTP, DHCP, and NFS Server. Has been running nicely for over a year. Security upgrade to latest kernel, kaboom! Tried upgrading to Buster. No good. As per https://github.com/raspberrypi/piserver/issues/75#issuecomment-516123335, blacklisted bcm2835_codec, bcm2835_v4l2, now stuff is booting.

Several different PI2 & PI3's. Raspian 4.19.66-v7+.

[gorky]

I wonder if https://github.com/raspberrypi/linux/issues/2528 has anything to do with this?

[trevorwarwick]

Argh... I've just spent 3 days trying to get remote boot working for the first time on a couple of Pis & suffering the same symptoms, and only just thought today to search for issues specific to buster which led me here. Good to have a workaround finally as I was totally stumped.