Using Bit Stream Filters

[hunterjm]

I am trying to convert an MP4 H264 stream to Annex-B for use in a MPEGTS output container. I am using the MP4 generated video file located in the docs under numpy for my reproduction example. I am getting an error saying that the stream is invalid, and needs to be in Annex B using -bsf:v h264_mp4toannexb

Error

H.264 bitstream malformed, no startcode found, use the video bitstream filter 'h264_mp4toannexb' to fix it ('-bsf:v h264_mp4toannexb' option with ffmpeg)

Reproduction code:

import io

import numpy as np
import av

duration = 4
fps = 24
total_frames = duration * fps

mp4 = io.BytesIO()
mp4.name = 'test.mp4'
container = av.open(mp4, mode='w')

stream = container.add_stream('libx264', rate=fps)
stream.width = 480
stream.height = 320
stream.pix_fmt = 'yuv420p'

for frame_i in range(total_frames):

    img = np.empty((480, 320, 3))
    img[:, :, 0] = 0.5 + 0.5 * np.sin(2 * np.pi * (0 / 3 + frame_i / total_frames))
    img[:, :, 1] = 0.5 + 0.5 * np.sin(2 * np.pi * (1 / 3 + frame_i / total_frames))
    img[:, :, 2] = 0.5 + 0.5 * np.sin(2 * np.pi * (2 / 3 + frame_i / total_frames))

    img = np.round(255 * img).astype(np.uint8)
    img = np.clip(img, 0, 255)

    frame = av.VideoFrame.from_ndarray(img, format='rgb24')
    for packet in stream.encode(frame):
        container.mux(packet)

# Flush stream
for packet in stream.encode():
    container.mux(packet)

# Close the file
container.close()


# Convert to mpegts
input_ = av.open(mp4, 'r')
input_stream = input_.streams.video[0]
output_file = io.BytesIO()
output_file.name = 'test.ts'
output = av.open(output_file, 'w', format='mpegts')
output_stream = output.add_stream('h264', input_stream.rate)

for packet in input_.demux(input_stream):
    # We need to skip the "flushing" packets that `demux` generates.
    if packet.dts is None:
        continue

    # We need to assign the packet to the new stream.
    packet.stream = output_stream

    output.mux(packet)

output.close()

[jlaine]

Hi Jason!

I had a dig around and it looks we don't currently have any code to deal with bitstream filters.

Unless I'm mistaken the API doesn't look very complicated so it might not be a huge deal to add this. The relevant functions seem to be av_bsf_*:

https://ffmpeg.org/doxygen/4.0/group__lavc__misc.html

Once the filter has been initialized it seems like it's "just" a matter of passing packets in using av_bsf_send_packet and retrieving the output by repeatedly calling av_bsf_receive_packet.

[hunterjm]

Hey @jlaine - There is a bitstream branch where @mikeboers has already started work on this, but it's created off of something other than develop. I started looking into this, but abandoned it in favor of going another direction. I still think it would be a useful feature to add though, so I'll keep the ticket open unless you want to close it.

[jlaine]

Yes, let's keep the ticket open, I'll take a look at Mike's branch. It's very handy you provided some self-contained test code, as it can form the base of a unit test.

[mikeboers]

The single commit on the bitstream branch should be easily cherry-pickable onto develop.

[mikeboers]

I've rebased this as a single commit onto develop.

[notedit]

Hi @mikeboers will we merge bitstream filter to develop?

[hunterjm]

@jlaine @mikeboers - As you might be able to see from the reference above, it looks like I might have a use case for this again. It looks like the branch still exists, but would definitely require a rebase. Is it viable to pick this back up for a future release?

[rarzumanyan]

Hi @mikeboers and @jlaine

What are the chances this patch will be merged to master and if it's possible how can I help you with that?

Bitstream filters support turned out to be extremely useful for a project which provides Python bindings to video decoding / encoding accelerated by Nvidia GPUs: https://github.com/NVIDIA/VideoProcessingFramework

It allows to extract Annex.B NAL Units and send them to GPU for HW - accelerated decoding (https://github.com/NVIDIA/VideoProcessingFramework/issues/99#issuecomment-744408969). Having this feature in master branch (and hence available with pip and such) would be great.

[philipp-schmidt]

+1 for this use-case

[brederle]

What is lacking for this feature to merge bitstream filters with main branch? I also require the h.264 annex bistream filter for my webrtc doorstation project. Thus, I am willing to support to get this feature from the library instead of wrapping it on my own.

[riaqn]

What is lacking for this feature to be merged to the main branch? really need it.

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[jlaine]

Let's keep this alive since it has a PR.

[tsdicloud]

I included it last month in my doorstation project, with a manual merge of the branch to current head. Testcases still work, but I am still not far enough to fully test the functionality.

The patch is especially useful if you get a half encoded x264 stream from raspberry cam and want to stream it to a webrtc solution.

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[skeskinen]

Since the PR #565 was closed, can this issue be opened again?

I also happen to have a usecase where I would need h264_mp4toannexb and hevc_mp4toannexb

@WyattBlue on the PR you said that you support this feature. Will you help me get bitstream filters merged if I try to rewrite the 4 year old code to match the current codebase?

I kind of started that already and it doesn't look too difficult. I'm hoping I'll have something working tomorrow.