Add grow method to Packet class, implementing av_grow_packet ffmpeg function

[MihanEntalpo]

Overview

I'm working with video streams from a hardware device, which adding SEI NAL-units with specific data, in slightly malformed format. ffmpeg complains about some problems of this format, but working fine.

I'm reading this NAL-units by programmatically parsing Packet bytes for NAL-units, finding the needed one, and reading data from it, as normal methods of working with side_data is not usable in my case.

Now, I need to be able to create such streams by myself for testing purposes, so I'm writing script to convert any mp4 (h264) file into file with such NAL-units.

Solution is simple: demux, add NAL-unit to package, mux to new stream.

But: I cannot add NAL-unit to package as it size is fixed. So, I need to have function grow on Packet class, what it implementing existing ffmpeg functionality.

Existing FFmpeg API

ffmpeg has function av_grow_packet that can increase size of packet by specific amount: https://ffmpeg.org/doxygen/trunk/group__lavc__packet.html#ga74f66e072998b8ce81ef3aba8d617a58

Expected PyAV API

So, pyav should just expose function Packet.grow(self, grow_by: int) what would do the trick.

Example:

# open source and destination files
with av.open("source.mp4", "r") as src_container:
    with av.open("dest.mp4", "w") as dest_container:
        # create video stream for destination file
        new_stream = dest_container.add_stream(template=src_container.streams.video[0])
        # walking through the packets
        for packet in src_container.demux(container.streams.video[0]):
            # changing packet stream
            packet.stream = new_stream
            # enlarging package by 16 bytes (the size of new NAL-unit)
            packet.grow(16)
            # appending packege with new NAL-unit, hardcoded for simplicity, actually would be generated
            packet.update(bytes(packet) + b"\0\0\0\0\0\0\1\0\6\6\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"
            # muxing packet
            dest_container.mux(packet)

Investigation

I've written small snippet on C++ with using av_grow_packet, and it worked, but I don't have enough knowledge to solve many problems that arise in C++, and doesn't in your wonderful python library

Reproduction

nope

Versions

• OS: Ubuntu 18.04.6 LTS
• PyAV runtime:

PyAV v11.0.1
library configuration: --disable-static --enable-shared --libdir=/tmp/vendor/lib --prefix=/tmp/vendor --disable-alsa --disable-doc --disable-mediafoundation --enable-fontconfig --enable-gmp --enable-gnutls --enable-gpl --enable-libaom --enable-libass --enable-libbluray --enable-libdav1d --enable-libfreetype --enable-libmp3lame --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libopenjpeg --enable-libopus --enable-libspeex --enable-libtheora --enable-libtwolame --enable-libvorbis --enable-libvpx --enable-libx264 --enable-libx265 --enable-libxcb --enable-libxml2 --enable-libxvid --enable-lzma --enable-version3 --enable-zlib
library license: GPL version 3 or later
libavcodec     59. 37.100
libavdevice    59.  7.100
libavfilter     8. 44.100
libavformat    59. 27.100
libavutil      57. 28.100
libswresample   4.  7.100
libswscale      6.  7.100

• FFmpeg:

ffmpeg version 3.4.11-0ubuntu0.1 Copyright (c) 2000-2022 the FFmpeg developers
built with gcc 7 (Ubuntu 7.5.0-3ubuntu1~18.04)
configuration: --prefix=/usr --extra-version=0ubuntu0.1 --toolchain=hardened --libdir=/usr/lib/x86_64-linux-gnu --incdir=/usr/include/x86_64-linux-gnu --enable-gpl --disable-stripping --enable-avresample --enable-avisynth --enable-gnutls --enable-ladspa --enable-libass --enable-libbluray --enable-libbs2b --enable-libcaca --enable-libcdio --enable-libflite --enable-libfontconfig --enable-libfreetype --enable-libfribidi --enable-libgme --enable-libgsm --enable-libmp3lame --enable-libmysofa --enable-libopenjpeg --enable-libopenmpt --enable-libopus --enable-libpulse --enable-librubberband --enable-librsvg --enable-libshine --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libssh --enable-libtheora --enable-libtwolame --enable-libvorbis --enable-libvpx --enable-libwavpack --enable-libwebp --enable-libx265 --enable-libxml2 --enable-libxvid --enable-libzmq --enable-libzvbi --enable-omx --enable-openal --enable-opengl --enable-sdl2 --enable-libdc1394 --enable-libdrm --enable-libiec61883 --enable-chromaprint --enable-frei0r --enable-libopencv --enable-libx264 --enable-shared
libavutil      55. 78.100 / 55. 78.100
libavcodec     57.107.100 / 57.107.100
libavformat    57. 83.100 / 57. 83.100
libavdevice    57. 10.100 / 57. 10.100
libavfilter     6.107.100 /  6.107.100
libavresample   3.  7.  0 /  3.  7.  0
libswscale      4.  8.100 /  4.  8.100
libswresample   2.  9.100 /  2.  9.100
libpostproc    54.  7.100 / 54.  7.100

Additional context

I'm going to fork your repo, and try to create this functionality by myself, as I'm already need it, maybe you can tell me what pitfalls should I avoid, especially considering that I have never written on Pyrex yet :)

[MihanEntalpo]

Update: I've managed to work around this problem. There are no av_grop_packet function exported, but I can create new packet from old packet's data.

So, my code is:

# open source and destination files
with av.open("source.mp4", "r") as src_container:
    with av.open("dest.mp4", "w") as dest_container:
        # create video stream for destination file
        new_stream = dest_container.add_stream(template=src_container.streams.video[0])
        # walking through the packets
        for packet in src_container.demux(container.streams.video[0]):
            # this is NALu I want to add:
            nalu = b"\0\0\0\0\0\0\1\0\6\6\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"
            # creating new packet 
            new_packet = av.Packet(input=bytes(packet) + nalu)
            # copy some fields values to new packet:
            for field in ["pts", "dts", "time_base"]:
                setattr(new_packet, field, getattr(packet, field))
            new_packet.stream = new_stream
            # muxing packet
            dest_container.mux(new_packet)

I have some problems with NALu format, and first key frame didn't go to the resulting file, but at first sight this method is working. The Grow method would be needed to make this operations faster, as creating new packets in python would be quite slow

[WyattBlue]

I don't think the spedup would be significant enough to warrant adding this feature. If anyone is intereseting in this feature, please reply here.