AV1Payloader does not work for WebRTC

[aleksitto-gh]

Your environment.

• Version: v1.8.6
• Browser: Google Chrome, Chromium, Arc

What did you do?

I'm trying to use AV1 codec for WebRTC using the libaom lib. The encoding works fine but probably AV1Payloader is not ready for that yet.

As a result, there is no video in my browser.

I found the browser expects to see obu size info but it is not sent.

[56649:64259:0703/185801.600480:WARNING:video_rtp_depacketizer_av1.cc(313)] Mismatch in obu_size. signaled: 0, actual: 1442
[56649:64259:0703/185801.637214:WARNING:video_rtp_depacketizer_av1.cc(313)] Mismatch in obu_size. signaled: 0, actual: 394
[56649:64259:0703/185801.664048:WARNING:video_rtp_depacketizer_av1.cc(313)] Mismatch in obu_size. signaled: 0, actual: 686
[56649:64259:0703/185801.701495:WARNING:video_rtp_depacketizer_av1.cc(313)] Mismatch in obu_size. signaled: 0, actual: 656
[56649:64259:0703/185801.732393:WARNING:video_rtp_depacketizer_av1.cc(313)] Mismatch in obu_size. signaled: 0, actual: 2044
[56649:64259:0703/185801.769189:WARNING:video_rtp_depacketizer_av1.cc(313)] Mismatch in obu_size. signaled: 0, actual: 276

The obu size flag is set as required via libaom in the obu header, I guess it is a default behavior.

I tried to manually change it as not required for the first encoded byte:

sMask := byte(0b11111101)
encoded[0] &= sMask

Packet buffer logic with the manually changed configuration:

pkt := C.aom_codec_get_cx_data(e.codec, &iter)
if pkt == nil {
break
}

if pkt.kind == C.AOM_CODEC_CX_FRAME_PKT {
...
pktBuf := C.aom_pktBuf(pkt)
pktSize := C.aom_pktSz(pkt)

encoded := C.GoBytes(unsafe.Pointer(pktBuf), pktSize)

sMask := byte(0b11111101)
encoded[0] &= sMask
}

c++ additional functions:

aom_codec_frame_flags_t aom_pktFrameFlags(aom_codec_cx_pkt_t *pkt) {
  return pkt->data.frame.flags;
}
void *aom_pktBuf(aom_codec_cx_pkt_t *pkt) {
  return pkt->data.frame.buf;
}
int aom_pktSz(aom_codec_cx_pkt_t *pkt) {
  return pkt->data.frame.sz;
}

it helped a little bit, the error with obu size mismatch has gone, but I found another errors on the browser side:

[56649:64259:0703/185901.432737:WARNING:video_receive_stream2.cc(831)] No decodable frame in 209975 us requesting keyframe. Last RTP timestamp 473597759.
[56649:64259:0703/185901.653905:WARNING:video_receive_stream2.cc(831)] No decodable frame in 219832 us requesting keyframe. Last RTP timestamp 473615438.
[56649:64259:0703/185901.876031:WARNING:video_receive_stream2.cc(831)] No decodable frame in 220216 us requesting keyframe. Last RTP timestamp 473635687.
[56649:64259:0703/185902.107335:WARNING:video_receive_stream2.cc(831)] No decodable frame in 220999 us requesting keyframe. Last RTP timestamp 473657668.
[56649:64259:0703/185902.322845:WARNING:video_receive_stream2.cc(831)] No decodable frame in 214457 us requesting keyframe. Last RTP timestamp 473675091.
[56649:64259:0703/185902.540952:WARNING:video_receive_stream2.cc(831)] No decodable frame in 216235 us requesting keyframe. Last RTP timestamp 473696303.
[56649:64259:0703/185902.767741:WARNING:video_receive_stream2.cc(831)] No decodable frame in 222425 us requesting keyframe. Last RTP timestamp 473718583.
[56649:64259:0703/185902.979717:WARNING:video_receive_stream2.cc(831)] No decodable frame in 210875 us requesting keyframe. Last RTP timestamp 473735219.

And still, no video is shown.

What did you expect?

AV1 Payload supports WebRTC. Video stream is playing correctly.

What happened?

• AV1Payloader does not support obu size info configs for OBU payloads
• Not sure exactly but maybe when obu size info is missing the browser can not understand when the keyframe is triggered because this info is not reflected in the AV1Payloader logic.

[lebedyncrs]

@aleksitto-gh tried this one ?

[aleksitto-gh]

@lebedyncrs yes, but it did not work for WebRTC

[JoeTurki]

Hello, I'm working on AV1 support and creating a Docker image to test Pion AV1 with various AV1 encoders and WebRTC clients.

Currently, I'm working on adding tests for libaom and dav1d. If you share your code I'm going to add it to the image. Thanks.

[3DRX]

Hi there, I'm using the updated version of pion/rtp and pion/webrtc with av1_nvenc and I found that the AV1 track can't be decoded correctly by chromium on linux (only the first frame is decoded), but on windows, chromium works. I've put more detail in #299 , not sure if this is a platform-specific bug in chromium.

[3DRX]

Hello, I'm working on AV1 support and creating a Docker image to test Pion AV1 with various AV1 encoders and WebRTC clients.

Currently, I'm working on adding tests for libaom and dav1d. If you share your code I'm going to add it to the image. Thanks.

I'd love to share my code using av1_nvenc with ffmpeg for the test if needed, currently the repo is private, I'll make it public after fixing a few issues.