fix: FIT-31: Ensure media buffers are syncable events that keep audio and video in sync

[bmartel]

Previously we were not syncing when a media source would buffer, which led to out of sync audio and video in many cases, especially in spotty network conditions or with very large file sizes. This addresses that issue by syncing buffering state events into SyncManager, and handling the state in the particular cases Audio, Video, Paragraphs. Also updated and aligned the buffering visuals by moving that into the TimelineControls.

This video showcases a ~300MB video throttled to 3G then seeked and played, and later unthrottled to simulate what was previously causing media sources to go out of sync with one another:

Kapture 2025-05-28 at 08.47.55.webm

Feature Flag:

• Added a new feature flag FF_SYNCED_BUFFERING to enable synchronized media buffering across components. (web/libs/core/src/lib/utils/feature-flags/flags.ts)

UI Updates:

• Introduced a visual buffering indicator in the Timeline component's controls, including new SCSS styles and animations. (web/libs/editor/src/components/Timeline/Controls.scss) [1] [2]
• Updated the VideoCanvas component to conditionally display a buffering indicator based on the buffering state. (web/libs/editor/src/components/VideoCanvas/VideoCanvas.scss, web/libs/editor/src/components/VideoCanvas/VideoCanvas.tsx) [1] [2]

Synchronized Buffering Logic:

• Added a useSyncedBuffering hook to manage synchronized buffering states across components using Jotai atoms. (web/libs/editor/src/hooks/useSyncedBuffering.ts)
• Updated the VideoCanvas and Html5Player components to utilize the synchronized buffering logic, including debounced state updates and conditional behavior based on the feature flag. (web/libs/editor/src/components/VideoCanvas/VideoCanvas.tsx, web/libs/editor/src/lib/AudioUltra/Controls/Html5Player.ts) [1] [2]

Playback Logic Enhancements:

• Modified the Player class to include a buffering property and handle buffering events (canplay and waiting) with debounced updates. (web/libs/editor/src/lib/AudioUltra/Controls/Player.ts) [1] [2]
• Adjusted the Timeline and VideoCanvas components to propagate buffering states through their props and interfaces. (web/libs/editor/src/components/Timeline/Types.ts, web/libs/editor/src/components/Timeline/Timeline.tsx) [1] [2]

These changes collectively improve the user experience by providing better visual feedback during buffering and ensuring consistent playback behavior across components.

[netlify[bot]]

✅ Deploy Preview for heartex-docs canceled.

Name	Link
🔨 Latest commit	4a4ed265865488c82b9941a282919a1cb09f4ffd
🔍 Latest deploy log	https://app.netlify.com/projects/heartex-docs/deploys/68a3581079cc9900084e32b8

[netlify[bot]]

✅ Deploy Preview for label-studio-storybook ready!

Name	Link
🔨 Latest commit	4a4ed265865488c82b9941a282919a1cb09f4ffd
🔍 Latest deploy log	https://app.netlify.com/projects/label-studio-storybook/deploys/68a358101514690008796690
😎 Deploy Preview	https://deploy-preview-7633--label-studio-storybook.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

---

To edit notification comments on pull requests, go to your Netlify project configuration.

[netlify[bot]]

✅ Deploy Preview for label-studio-playground ready!

Name	Link
🔨 Latest commit	4a4ed265865488c82b9941a282919a1cb09f4ffd
🔍 Latest deploy log	https://app.netlify.com/projects/label-studio-playground/deploys/68a35810a8ed590008883dfd
😎 Deploy Preview	https://deploy-preview-7633--label-studio-playground.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

---

To edit notification comments on pull requests, go to your Netlify project configuration.

[netlify[bot]]

✅ Deploy Preview for label-studio-docs-new-theme canceled.

Name	Link
🔨 Latest commit	4a4ed265865488c82b9941a282919a1cb09f4ffd
🔍 Latest deploy log	https://app.netlify.com/projects/label-studio-docs-new-theme/deploys/68a3581027162300080260fb

[bmartel]

/fm sync

Workflow run

[codecov[bot]]

Codecov Report

:x: Patch coverage is 90.00000% with 53 lines in your changes missing coverage. Please review. :white_check_mark: Project coverage is 66.61%. Comparing base (2ba7706) to head (4a4ed26). :warning: Report is 20 commits behind head on develop.

Files with missing lines	Patch %	Lines
web/libs/editor/src/tags/object/Audio/view.tsx	84.25%	17 Missing :warning:
.../editor/src/components/VideoCanvas/VideoCanvas.tsx	71.69%	15 Missing :warning:
web/libs/editor/src/tags/object/Audio/model.js	94.28%	4 Missing :warning:
...ditor/src/tags/object/Paragraphs/HtxParagraphs.jsx	84.21%	3 Missing :warning:
.../libs/editor/src/lib/AudioUltra/Controls/Player.ts	93.75%	2 Missing :warning:
...ibs/editor/src/lib/AudioUltra/Visual/Visualizer.ts	86.66%	2 Missing :warning:
web/libs/editor/src/lib/AudioUltra/Waveform.ts	50.00%	2 Missing :warning:
...eb/libs/editor/src/tags/object/Paragraphs/model.js	97.18%	2 Missing :warning:
web/libs/editor/src/tags/object/Video/HtxVideo.jsx	84.61%	2 Missing :warning:
web/libs/editor/src/hooks/useUpdateBuffering.ts	95.65%	1 Missing :warning:
... and 3 more

Additional details and impacted files

@@             Coverage Diff             @@
##           develop    #7633      +/-   ##
===========================================
- Coverage    70.43%   66.61%   -3.83%     
===========================================
  Files          721      509     -212     
  Lines        52231    34576   -17655     
  Branches      8837     8990     +153     
===========================================
- Hits         36789    23032   -13757     
+ Misses       15439    11541    -3898     
  Partials         3        3              

Flag	Coverage Δ
lsf-e2e	60.03% <64.83%> (+0.32%)	:arrow_up:
lsf-integration	55.98% <83.93%> (+0.44%)	:arrow_up:
lsf-unit	9.41% <5.78%> (-0.02%)	:arrow_down:
pytests	?

Flags with carried forward coverage won't be shown. Click here to find out more.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

:rocket: New features to boost your workflow:

• :package: JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[Gondragos]

/git merge

Workflow run Successfully merged: create mode 100644 web/libs/editor/src/components/TaskSummary/tests/TaskSummary.test.tsx

[niklub]

/git merge

Workflow run Successfully merged: delete mode 100644 web/libs/app-common/src/blocks/StorageProviderForm/providers/localFiles.ts

[niklub]

/git merge

Workflow run Successfully merged: create mode 100644 label_studio/tests/data_import/test_streaming_import.py

[niklub]

/git merge

Workflow run Successfully merged: create mode 100644 web/libs/editor/src/lib/AudioUltra/Visual/Renderer/ResizeRenderer.ts

[niklub]

/git merge

Workflow run Successfully merged: delete mode 100644 web/libs/editor/src/regions/ImageRegion.js

[Gondragos]

/git merge

Workflow run Successfully merged: 79 files changed, 1663 insertions(+), 347 deletions(-)

[Gondragos]

/git merge

Workflow run Successfully merged: 7 files changed, 159 insertions(+), 20 deletions(-)