Playback from bag files - direct access to frames by index or one after the other

[CBadam]

Hello, I am recording the color and depth streams for offline processing that takes a long time. To reproduce the recorded sequence, I am following the playback sample. https://github.com/orbbec/pyorbbecsdk/blob/main/examples/playback.py. However, my objective is to process every frame of the sequence, and using the example code and adding a processing function after "frames = pipeline.wait_for_frames(100)", I am experiencing a drop of frames (especially when the processing takes a while). Thus, I was wondering if there is a way to access the frames by index, one after one and not as if it is recording in real time. something like this: playback.set_real_time( false ); from realsense Thanks in advance!

[hamamabrouk]

Description: I am encountering the same issue with the playback mode of the pyorbbecsdk.Pipeline when processing .bag files. During playback, the pipeline continuously streams frames without the ability to pause or control the playback. This results in frames being lost or skipped when trying to save RGB images, depth images, and point clouds in real time.

Expected Behavior: The pipeline should provide a way to pause or control the playback (e.g., manual stepping through frames). This would allow enough time to process and save each frame (such as saving RGB images, depth images, and generating point clouds) without missing or skipping frames due to continuous playback.

Current Behavior: The pipeline continues playing and doesn't pause when the frames are being saved, causing some frames to be skipped. There is no explicit method like pause() or set_playback_speed(0) to manually control the playback or pause the pipeline during playback. Any attempts to stop or slow down the pipeline interrupts the playback entirely, making it difficult to process every frame properly. Steps to Reproduce: Initialize the pyorbbecsdk.Pipeline in playback mode with a .bag file. Attempt to read frames and save each frame as an RGB image, depth image, and point cloud. Observe that frames are skipped while saving due to the continuous playback.

Proposed Solution: Provide a pause() method or a set_playback_speed(0) functionality to allow manual control over the frame playback. Ensure that the pipeline can be paused without fully stopping, so that frames can be processed and saved without missing any frames.

Environment: SDK Version: v1.10.8 Platform: Ubuntu 22.04 LTS Hardware: Orbbec Femto Mega Camera

[zhonghong322]

@hamamabrouk @CBadam Thank you for your suggestions. We will consider them in future versions of the Orbbec SDK to improve our recording and playback experience.

[David-Conrad]

Had the same problem/wish, but i just found out, that the frame-callback function seems to be blocking. So that only one callback function can be active at a time. This could enable a workaround, were you can put a Queue with blocking in between. At least this way you can iterate through the recording chronologically. Didn't test it yet, since my task is different now, but maybe I'll come back to this later. (Also I didn't test, wether there is a drop of images or not)

[natelowry]

looks like the V2 of the SDK has play/pause

docs for playback

might not be helpful as you can't read V1 .bag files w/ the V2 SDK

[natelowry]

update on this as we're migrating to the V2 SDK. playback does have play/pause, but the frame-callback function does not seem to be blocking on playback...so not sure what we'll end up doing. will try to post here when we get a workaround.

[natelowry]

tried some pretty gross hacks like doing a quick resume then an immediate pause, but the playback loop and its timing is...messy to say the least, so you can pretty easily accidentally advance 2 frames.

you could just queue all the frames in the callback and do your own pause/resume. good luck with the memory usage 🙃

this seems to be working for our use case:

1. track your current frame index by incrementing it in the frameset callback
2. start with the playback device paused
3. in your NextFrame function get the current index and loop until it changes (you have a new frame)
4. ^in that loop just keep calling resume on the playback device
5. in the frameset callback, immediately call pause on the playback device when you get a new frameset. this is the only way i've found to make sure it doesn't accidentally process the next frame.

i think you're still at risk of it skipping a frame as the underlying playback device actually queues all the frames and pumps the callback in a separate thread, but this might be enough to get you by...