pion
AV1Payloader is not compatible with av1-rtp-spec
Your environment.
- Version: v1.7.13
- Other Information:
The av1-rtp-spec has a lot of limitations, which is not handled by current implement of
AV1Payloader.Payloadhttps://github.com/pion/rtp/blob/9a6e1d08c3e7b8ec582c099f0071629f3bfe7356/codecs/av1_packet.go#L28
av1-rtp-spec
It requires obu element size in payload when W=0, and has limits for temporal_id and spatial_id in same packet, etc.
Current implement will results in invalid packet.
What did you do?
Trying to build a play-from-disk-ivf demo, which supports vp8, vp9 and av1 in ivf container.
What did you expect?
All codecs plays in browser. Or, show error in some means when input data is invalid. (The AV1Payloader.Payload do not return a error, but at least some log message to tell me somethin goes wrong here.)
What happened?
Browser failed to decode the AV1 packets, and no error message from pion. (See discussion here https://gophers.slack.com/archives/CAK2124AG/p1654875989280929)
After some investigation I found that the reason is the rtp packets are incompatible with av1-rtp-spec.
In my implement I just put the frame date into WriteSample, which uses AV1Payloader.Payload to packetize it for rtp.
Since there is no check or transform for input data, the result is invalid and it's hard to find the reason.
It seems the AV1Payloader.Payload will work if I always input a single obu, which is not what I get from IVFReader.
It will to be easier to handle one obu a time, I guess that's why current implement use this approach.
In this case, so some document or example should be good too.
I still thinks it will be better to change the AV1Payloader.Payload, so it will handle multiple obus too. I'll try to implement it, here is my thoughts: First, split the frame data into separate obus. If there are several obus with hasSizeFlag=1, just do the split and remove size inside obu data. And if there is a obu with hasSizeFlag=0, then treat all remaining data as obu data, so it is compatible with current behavior. Then, re-organize the obus according to the av1-rtp-spec.
Any more suggestions here?
The reason why AV1 packets packetized with this library can't be decoded by the browsers is that the N bit of the AV1 aggregation header is always set to zero, while it should be set to 1 in case of a first packet of a coded video sequence.
I have a working encoder here: https://github.com/bluenviron/gortsplib/blob/main/pkg/formats/rtpav1/encoder.go
and i'd like to contribute it entirely to pion/rtp but i can't do it until this library is frozen.
Sorry this took so long @AutumnSun1996 !
Pion's play-from-disk and save-to-disk-av1 now work together.
