how to screen capture with ffmpy

[ReenigneArcher]

I am attempting to use ffmpy and ffmpeg to screen capture the desktop, then send the frames to another device.

from ffmpy import FFmpeg
import numpy as np
import subprocess

ff = FFmpeg(
    executable=os.path.join('ffmpeg', 'bin', 'ffmpeg.exe'),
    inputs={'desktop': '-f gdigrab -framerate 30'},
    outputs={'pipe:1': '-c:v h264'}
)

pipe = ff.run(stdout=subprocess.PIPE)

while True:
    # read 1920*1080*3 bytes (= 1 frame)
    raw_image = pipe.stdout.read(1920 * 1080 * 3)
    # transform the byte read into a numpy array
    frame = np.fromstring(raw_image, dtype='uint8')
    # throw away the data in the pipe's buffer.
    pipe.stdout.flush()

I get the following:

(venv) C:\Users\ReenigneArcher\Documents\GitHub\RetroArcher\py_stream>python test.py
ffmpeg version n5.0.1-5-g240d82f26e-20220626 Copyright (c) 2000-2022 the FFmpeg developers
  built with gcc 11.2.0 (crosstool-NG 1.24.0.533_681aaef)
  configuration: --prefix=/ffbuild/prefix --pkg-config-flags=--static --pkg-config=pkg-config --cross-prefix=x86_64-w64-mingw32- --arch=x86_64 --target-os=mingw32 --enable-gpl --enable-version3 --disable-debug --disable-w32threads
 --enable-pthreads --enable-iconv --enable-libxml2 --enable-zlib --enable-libfreetype --enable-libfribidi --enable-gmp --enable-lzma --enable-fontconfig --enable-libvorbis --enable-opencl --disable-libpulse --enable-libvmaf --disa
ble-libxcb --disable-xlib --enable-amf --enable-libaom --enable-libaribb24 --enable-avisynth --enable-libdav1d --enable-libdavs2 --disable-libfdk-aac --enable-ffnvcodec --enable-cuda-llvm --enable-frei0r --enable-libgme --enable-l
ibass --enable-libbluray --enable-libmp3lame --enable-libopus --enable-librist --enable-libtheora --enable-libvpx --enable-libwebp --enable-lv2 --enable-libmfx --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libopen
h264 --enable-libopenjpeg --enable-libopenmpt --enable-librav1e --enable-librubberband --enable-schannel --enable-sdl2 --enable-libsoxr --enable-libsrt --enable-libsvtav1 --enable-libtwolame --enable-libuavs3d --disable-libdrm --d
isable-vaapi --enable-libvidstab --enable-vulkan --enable-libshaderc --enable-libplacebo --enable-libx264 --enable-libx265 --enable-libxavs2 --enable-libxvid --enable-libzimg --enable-libzvbi --extra-cflags=-DLIBTWOLAME_STATIC --extra-cxxflags= --extra-ldflags=-pthread --extra-ldexeflags= --extra-libs=-lgomp --extra-version=20220626
  libavutil      57. 17.100 / 57. 17.100
  libavcodec     59. 18.100 / 59. 18.100
  libavformat    59. 16.100 / 59. 16.100
  libavdevice    59.  4.100 / 59.  4.100
  libavfilter     8. 24.100 /  8. 24.100
  libswscale      6.  4.100 /  6.  4.100
  libswresample   4.  3.100 /  4.  3.100
  libpostproc    56.  3.100 / 56.  3.100
[gdigrab @ 000002c5622b2980] Capturing whole desktop as 1920x1080x32 at (0,0)
[gdigrab @ 000002c5622b2980] Stream #0: not enough frames to estimate rate; consider increasing probesize
Input #0, gdigrab, from 'desktop':
  Duration: N/A, start: 1656372396.709033, bitrate: 1990668 kb/s
  Stream #0:0: Video: bmp, bgra, 1920x1080, 1990668 kb/s, 30 fps, 1000k tbr, 1000k tbn
[NULL @ 000002c560479080] Unable to find a suitable output format for 'pipe:1'
pipe:1: Invalid argument
Traceback (most recent call last):
  File "C:\Users\ReenigneArcher\Documents\GitHub\RetroArcher\py_stream\test.py", line 31, in <module>
    pipe = ff.run(stdout=subprocess.PIPE)
  File "C:\Users\ReenigneArcher\Documents\GitHub\RetroArcher\py_stream\venv\lib\site-packages\ffmpy.py", line 106, in run
    raise FFRuntimeError(self.cmd, self.process.returncode, out[0], out[1])
ffmpy.FFRuntimeError: `ffmpeg5\bin\ffmpeg.exe -f gdigrab -framerate 30 -i desktop -c:v h264 pipe:1` exited with status 1

STDOUT:


STDERR:

This command works in command line.

ffmpeg -f gdigrab -framerate 30 -i desktop output.mkv

[Ch00k]

Looks like the command is not correct. The equivalent of what ffmpy is trying to run is ffmpeg5\bin\ffmpeg.exe -f gdigrab -framerate 30 -i desktop -c:v h264 pipe:1. And it actually says that in the output:

[NULL @ 000002c560479080] Unable to find a suitable output format for 'pipe:1'
pipe:1: Invalid argument

Try specifying the output format, like so: ffmpeg5\bin\ffmpeg.exe -f gdigrab -framerate 30 -i desktop -c:v h264 -f avi pipe:1

[ReenigneArcher]

Okay, thanks for taking the time to answer! Adding -f avi does get it to pipe the output to stdout. Any idea why avi works, but others like mp4 and mkv do not? Do you have any good references on ffmpeg command line? A lot of this stuff is lacking in the ffmpeg documentation.

Also, I want to read the frames in real time. I guess ffmpy doesn't have a subprocess.popen equivalent? The run function is blocking. It's okay if it doesn't I can just switch to the build in subprocess.

[Ch00k]

avi was just an example of how you should set an output format. Not sure why mp4 or mkv don't work for you, but there is probably a hint to that in ffmpeg's output if you try to run such a command in the terminal.

As for the documentation, in my opinion the best source is https://ffmpeg.org/ffmpeg-all.html

The FFmpeg.run is indeed blocking, but I guess you could try running your frame capturing code in a separate thread.