Video Artifacts Occur With SampleBuilder Reconstruction Until Significant Movement Occurs

[VerioN1]

What happened?

When using samplebuilder to reconstruct frames from RTP packets of a WebRTC video stream, visual artifacts appear after a few seconds of stable or low-motion video. These artifacts persist until there is a moment of significant movement in the video scene, at which point the frame alignment seems to correct itself, and the video quality returns to normal.

What did you do?

1. Set up a WebRTC video track using codecs like VP8, VP9, or H264.
2. Use samplebuilder.New(35, pkt, t.Codec().ClockRate) and push incoming RTP packets to it.
3. Pop samples from samplebuilder and write them directly to a video track (videoTrack.WriteSample()).
4. Observe the video output after a few seconds of minimal movement:
   • Video frames start showing pixelation and artifacts.
   • These artifacts remain until a sudden movement occurs in the video, after which the frames realign and the image quality returns to normal.

Expected Behavior: The samplebuilder should produce clean, artifact-free frames regardless of the motion level in the video. All frames should remain synchronized and properly decoded.

Actual Behavior: Frames become misaligned or partially corrupted over time in low-motion scenes, leading to visible artifacts. Only significant motion triggers a correction and returns the output to a stable, artifact-free state.

Code Example:

pc.OnTrack(func(t *webrtc.TrackRemote, receiver *webrtc.RTPReceiver) {
  if t.Codec().MimeType == webrtc.MimeTypeVP8 ||
     t.Codec().MimeType == webrtc.MimeTypeVP9 ||
     t.Codec().MimeType == webrtc.MimeTypeH264 {
  
      var pkt rtp.Depacketizer
      switch t.Codec().MimeType {
      case webrtc.MimeTypeVP8:
          pkt = &codecs.VP8Packet{}
      case webrtc.MimeTypeVP9:
          pkt = &codecs.VP9Packet{}
      case webrtc.MimeTypeH264:
          pkt = &codecs.H264Packet{}
      }
  
      builder := samplebuilder.New(35, pkt, t.Codec().ClockRate)
  
      for {
          select {
          case <-stop:
              fmt.Println("Pushed sample to sampleChan stop")
              return
          default:
              rtpPacket, _, err := t.ReadRTP()
              if err != nil {
                  fmt.Println("ReadRTP error:", err.Error())
                  return
              }
              builder.Push(rtpPacket)
              for sample := builder.Pop(); sample != nil; sample = builder.Pop() {
                  if err := videoTrack.WriteSample(*sample); err != nil {
                      fmt.Println("WriteSample error:", err.Error())
                      return
                  }
              }
          }
      }
  }
}

Additional Context:

• Adding a partition head checker or enabling proper frame boundary logic might resolve this problem.

Environment:

• Pion WebRTC version: v4 latest
• Operating System: (golang:1.23-bullseye)
• app: chrome and flutter-webrtc (ios)
• this is almost 1:1 copy of the reflect exmaple

example image :

the complete example be made from this repo: https://github.com/VerioN1/webrtc-poc-go/tree/main