YUYV from V4L panics with "img does not implement SubImage()" error

[natevw]

I'm hoping to get this working with a V4L (loopback) device fed by ffmpeg. If not for testing (since only a few trains a day at my location) but also so that I can eventually feed it from a networked camera via a go2rtc RTSP feed. But for now just with a sample recording of a train going past to see it work.

So in one terminal I do:

sudo modprobe v4l2loopback

export FEED_URL=~/sample.mp4
export OUTPUT=/dev/video0

ffmpeg -i "$FEED_URL" -f v4l2 -pix_fmt yuyv422 "$OUTPUT"

And then in another terminal I do:

export DATA_DIR=~/trains
export INPUT=/dev/video0
export CAMERA_W=2304
export CAMERA_H=1296
export RECT_X=1152
export RECT_Y=818
export RECT_W=473
export RECT_H=242
export PX_PER_M=19  # e.g. 430 px, 23 m
# see https://github.com/jo-m/trainbot/blob/43ad8c9716f385ef0714a5057dfe751360844e8d/pkg/vid/cam.go#L213
# and `V4L2_PIX_FMT_…` in https://www.kernel.org/doc/html/v4.10/media/uapi/v4l/videodev.html#videodev2-h
#export CAMERA_FORMAT_FOURCC=422P
#export CAMERA_FORMAT_FOURCC=mjpg
export CAMERA_FORMAT_FOURCC=YUYV

./trainbot

This combination gets me the farthest without general opening/format errors but then I am left with an error:

{"level":"panic","error":"img does not implement SubImage()","time":"2023-11-22T02:18:16.864Z","caller":"/src/cmd/trainbot/main.go:162","message":"failed to crop frame"}

Any advice?

[jo-m]

Can you send me one of your video files, in your exact pixel format? Then I can try to reproduce. What ffmpeg version do you have?

In general, as you might have noticed by looking at the code, the video input situation for Go is pretty bad. There are some v4l libraries, but they are pretty limited in features, and for both ffmpeg and raspicam, the "best" solution turned out to run them as binaries and read from their stdout..

[natevw]

Can you send me one of your video files, in your exact pixel format? Then I can try to reproduce. What ffmpeg version do you have?

I did paste attach some sample at https://github.com/jo-m/trainbot/issues/16#issuecomment-1823814941 although that one is heavily modified from the original, and my ffmpeg says:

ffmpeg version 4.3.6-0+deb11u1 Copyright (c) 2000-2023 the FFmpeg developers
  built with gcc 10 (Debian 10.2.1-6)
  configuration: --prefix=/usr --extra-version=0+deb11u1 --toolchain=hardened --libdir=/usr/lib/aarch64-linux-gnu --incdir=/usr/include/aarch64-linux-gnu --arch=arm64 --enable-gpl --disable-stripping --enable-avresample --disable-filter=resample --enable-gnutls --enable-ladspa --enable-libaom --enable-libass --enable-libbluray --enable-libbs2b --enable-libcaca --enable-libcdio --enable-libcodec2 --enable-libdav1d --enable-libflite --enable-libfontconfig --enable-libfreetype --enable-libfribidi --enable-libgme --enable-libgsm --enable-libjack --enable-libmp3lame --enable-libmysofa --enable-libopenjpeg --enable-libopenmpt --enable-libopus --enable-libpulse --enable-librabbitmq --enable-librsvg --enable-librubberband --enable-libshine --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libsrt --enable-libssh --enable-libtheora --enable-libtwolame --enable-libvidstab --enable-libvorbis --enable-libvpx --enable-libwavpack --enable-libwebp --enable-libx265 --enable-libxml2 --enable-libxvid --enable-libzmq --enable-libzvbi --enable-lv2 --enable-omx --enable-openal --enable-opencl --enable-opengl --enable-sdl2 --enable-pocketsphinx --enable-libdc1394 --enable-libdrm --enable-libiec61883 --enable-chromaprint --enable-frei0r --enable-libx264 --enable-shared
  libavutil      56. 51.100 / 56. 51.100
  libavcodec     58. 91.100 / 58. 91.100
  libavformat    58. 45.100 / 58. 45.100
  libavdevice    58. 10.100 / 58. 10.100
  libavfilter     7. 85.100 /  7. 85.100
  libavresample   4.  0.  0 /  4.  0.  0
  libswscale      5.  7.100 /  5.  7.100
  libswresample   3.  7.100 /  3.  7.100
  libpostproc    55.  7.100 / 55.  7.100

but I don't think that's the problem because in this report I am not using it directly with your golang code. Because ffmpeg was being used only to feed into the loopback /dev/video0 which is read by golang as any other V4L device I would expect, and unaware of ffmpeg and original format?

for both ffmpeg and raspicam, the "best" solution turned out to run them as binaries and read from their stdout..

I actually wonder if all could be done with ffmpeg alone? Could it read in from any of its supported formats (e.g. including not just local file, but also I believe it supports V4L itself as well as things like RTSP and other webcam feed URLs) and maybe even -vf "crop=…" (docs) to just get only the needed pixels from it?

Because what I've been doing as a workaround for now since my camera situation is still somewhat changing and testing is just recording a few samples as I am able and then playing those from files which does work — and that's been working (so long as I pre-crop to avoid #19) as far as not getting input/format/SubImage() errors!

[jo-m]

I actually wonder if all could be done with ffmpeg alone? Could it read in from any of its supported formats (e.g. including not just local file, but also I believe it supports V4L itself as well as things like RTSP and other webcam feed URLs) and maybe even -vf "crop=…" (docs) to just get only the needed pixels from it?

Yes. This is exactly what I am doing with the libcamera implementation (for the Raspberry Pi Cam 3): https://github.com/jo-m/trainbot/blob/03bcf15b035c385913321b2b87c3f40eaae5a28c/pkg/vid/picam3.go#L86

When I started the project, I was using a v4l webcam via github.com/vladimirvivien/go4vl (pkg/vid/cam.go), so I had to do all the copping etc myself in Go. Then, I started using ffmpeg a lot for testing with recorded videofiles (pkg/vid/vid.go) - and I kept the old code which does the cropping in Go. In the end, I settled with a raspi cam for camera HW, so I implemented the libcamera source (pkg/vid/picam3.go). Currently, the v4l code is used for /dev/videoX, ffmpeg for anything that looks like a video file, and picam3 if you pass the literal string 'picam3'.

It would not be too hard to change the ffmpeg source to 1. be used for v4l etc as well and 2. dirctly implement cropping. In that case, I'd perhaps change the ffmpeg src to also accept the same struct as the Pi cam for config (https://github.com/jo-m/trainbot/blob/03bcf15b035c385913321b2b87c3f40eaae5a28c/pkg/vid/picam3.go#L35).

[natevw]

So yes, I'd love if basically I could pass basically any source (and options) to ffmpeg and then I suppose trainbot would just tack on whatever final rawvideo-type arguments it needs to access the final result. Then I could just grab straight from an RTSP feed and maybe even apply some rotation if it can keep up CPU-wise. But perhaps I should file a separate feature request in case this issue is getting too far off-topic?

Was hoping to take a swing at routing more input options into ffmpeg myself, but it's been awhile since I did much golang development and with this repo even if I try download the modules first attempting:

go env -w GO111MODULE=on
go mod download

within a checkout of this repo I still get a bunch of errors from go build -o build/trainbot ./cmd/trainbot like:

internal/pkg/db/db.go:6:2: package embed is not in GOROOT (/usr/lib/go-1.15/src/embed)
../go/pkg/mod/go-hep.org/x/[email protected]/hplot/hplot_gen.go:10:2: import "github.com/campoy/embedmd" is a program, not an importable package
pkg/ransac/ransac.go:11:2: found packages fit (curve1d.go) and main (run.go) in /home/debian/go/pkg/mod/go-hep.org/x/[email protected]/fit
../go/pkg/mod/golang.org/x/[email protected]/tiff/reader.go:20:2: import "golang.org/x/image/ccitt" is a program, not an importable package
…snip…

using go version go1.15.15 linux/arm64 available in the Debian 11 install I had handy.

[jo-m]

So yes, I'd love if basically I could pass basically any source (and options) to ffmpeg and then I suppose trainbot would just tack on whatever final rawvideo-type arguments it needs to access the final result. Then I could just grab straight from an RTSP feed and maybe even apply some rotation if it can keep up CPU-wise. But perhaps I should file a separate feature request in case this issue is getting too far off-topic?

This would be pretty easy, and 90% of the work for that would already be done, in https://github.com/jo-m/trainbot/blob/master/pkg/vid/vid.go.

As for your building problems, my guess would be that you need a newer go version. I am using go 1.21 from a PPA: https://launchpad.net/~longsleep/+archive/ubuntu/golang-backports