Buffered-Video-Stream Sample Application

[stefankiesz]

Issue #, if available:

Description of Changes

Summary

• Added a new sample application to demonstrate how one may keep a local buffer of media for a specified duration to be streamed to KVS upon a camera event occurring.

The use-case this application captures is event-based streaming to KVS that includes some footage leading up to the camera event. Once an event is triggered, the application streams frames from a specified amount of time prior to the event which have been kept in a local memory buffer, and begins live-streaming all new frames until the event duration has been reached. Once the event is "over", the application stops live streaming and instead goes back to writing frames to a local buffer. When the local buffer is full (buffer duration reached), the oldest frames are dropped to make way for the latest.

In order to maintain group-of-picture (GoP) validity, we cannot drop from the buffer I-frames nor subsequent P-frames that are associated (of the same GoP) with a later P-frame that is to be kept on the buffer. For this reason, the application organizes frames into GoP objects that contain an I-frame and a list of associated P-frames. The application's video buffer is a list of such GoP objects.

So, in the case that the specified buffer duration boundary lays in the middle of a GoP, the application will not drop any frames from the GoP, thus the actual buffered video will be longer than the specified buffer duration.

Another characteristic of this video buffer implementation is that, in order to reduce latencies in-sync with the GStreamer frames stream, the application only checks for old GoPs to discard on every I-frame/key-frame it receives from the incoming camera stream. This adds an additional extra GoP of frames that may be kept longer than the specified buffer duration. This extra GoP would be held in the buffer in the case such that a buffered GoP contains a P-frame that lays on the buffer duration boundary, and the frame received from the camera upon when the GoP completely falls off from the buffer duration is not an I-frame. Once the last P-frame of that GoP is no longer within the buffer duration, we should drop the GoP, but we won't until we receive an I-frame from the incoming stream, so if we don't happen to be on a new I-frame when that now-expired GoP falls off, we will hold on to it for a little longer than we need.

The buffer duration being slightly longer than the exact duration specified is preferred over:

• The possibility for buffered P-frames to be invalid due to no associated I-frame.
  • This would actually then reduce the buffer duration to be bellow the desired duration.
• The possibility of the GStreamer pipeline being backed up due to congestion of frames if we were to check for expired GoPs every single frame.

TODO:

• Add a section to the ReadMe.
• Add CI tests
• Fix GetMedia playback

Testing

Frequent Events, Short Buffer Duration, Buffered-to-Live Transition in Middle of Fragment

This test used the below sample application settings:

#define CAMERA_EVENT_LIVE_STREAM_DURATION_SECONDS 3
#define CAMERA_EVENT_COOLDOWN_SECONDS             9
#define STREAM_BUFFER_DURATION_SECONDS            3

Viewing the ingested stream on a visual timeline shows that there is a consistent pattern of media present (indicated by the orange bars) and media missing (these are the gaps in the orange bar that were not streamed to KVS because they were outside of the buffer's duration). In this timeline view, the orange bar is broken up into fragments which are indicated by the black borders around pieces of the orange bars. To the right of the timeline is a fragment browser that shows a thumbnail and extra details of each of the fragments.

Given the above application parameters, we expect ingested media to be present for CAMERA_EVENT_LIVE_STREAM_DURATION_SECONDS + STREAM_BUFFER_DURATION_SECONDS(6) seconds, and for there to be gaps of CAMERA_EVENT_COOLDOWN_SECONDS - STREAM_BUFFER_DURATION_SECONDS(6) seconds. When viewing the ingested stream on a visual timeline, the ratio of media to gaps should be 1:1, but it is closer to 2:1, with their being about 8 seconds of media to 4 seconds of no media. This is due to the head-end (oldest) of the buffer duration being a P-frame associated with an older I-frame; we must not discard the I-frame in order for its subsequent P-frames to be decodable. Below, we will further validate that this is the case.

Logs show that the buffered-to-live transition happens between frames with timestamps 4:53:37.561 and 4:53:37.594 (PM GMT-07:00 DST):

[DEBUG] [18-03-2025 23:53:37:878.570 UTC] Putting buffered frame with ts: 17423420175614330
[INFO ] [18-03-2025 23:53:37:878.605 UTC] Done sending buffered frames...
[DEBUG] [18-03-2025 23:53:37:878.617 UTC] Putting live frame with ts: 1742342017594766333

Looking for that 4:53:37... timestamp in the ingested KVS fragments, we see this transition occurs in the middle of a ~4-sec-long fragment (the timestamp displayed below the cyan playhead shows the cursor-hovered-over fragment's start time):

The playback across the transition is smooth with no interruptions.

We can also check the integrity of the start and end times of the overall event stream.

The first frame of the first fragment has timestamp 4:53:32.461:

The final fragment has a start time of 4:53:40.461 and a duration of ~0.1sec:

So the total clip length associated with this event is 8.1 seconds.

And looking at the first fragment associated with the next stream event, we see that it's first frame has the timestamp 4:53:44.461,

which shows that the duration of the gap between events is 3.9 seconds.

Maxing-Out the Content Store

Using the sample's default KVS_CONTENT_STORE_SIZE_BYTES of 128 MB and devicesrc, the STREAM_BUFFER_DURATION_SECONDS was increased until the KVS Producer started to drop old, not-yet-ingested, frames from its internal content store (frames go on the content store after putFrame is called on them). The maximum buffer duration achieved before KVS Producer started to drop the oldest frames was 1 minute and 44 seconds:

Memory Growth over Time

The RSS memory used by the application was checked before and after running for 12+ hours. Here is the command used to check the RSS utilization on Mac:

( echo "PID COMMAND RSS VSZ %MEM %CPU" ; ps -eo pid,command,rss,vsz,%mem,%cpu | grep -i "KVS" | sort -k3 -nr ) | column -t

As the buffer filled up, memory utilization grew until a camera event was triggered. Once the buffered was cleared after the frames were streamed, memory returned to the levels before the buffer was populated. This maximum and minimum memory utilization remained the same before and after the 12+ hours.

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.