[Feature Request] Skip the silence in media playback

[truthinhoney]

Is your feature request related to a usage problem (not a bug)? Please describe. No, this feature request is not related to any bugs or usage problems.

Describe the solution you'd like to see implemented There is a unique feature in Google ExoPlayer (used by NewPipe) that allows the viewer to fast forward and skip the silence in media playback. I would like to have a similar feature in FreeTube that is toggled on and off somewhere.

Describe alternatives you've considered I struggled to find any media player that has a similar feature.

Screenshots N/A

Additional context I have provided a link below for the relevant code in ExoPlayer that accomplishes this task.

http://github.com/google/ExoPlayer/blob/release-v2/library/core/src/main/java/com/google/android/exoplayer2/audio/SilenceSkippingAudioProcessor.java

[peepo5]

What would skipping silence be used for ?

[truthinhoney]

What would skipping silence be used for ?

The main purpose of skipping silence is to reduce the overall time spent consuming media.

Skipping silence is more gentle alternative to directly fast forwarding the playback speed.

The beauty of skipping silence is its subtlety, I feel comfortable to always keep it enabled because it will not be triggered when switching to a music video from a lecture.

This feature is particularly useful when consuming media that does not have background music, such as academic lectures, broadcast news, or podcasts.

[peepo5]

No I know what skipping silence does, I am saying how does it work? When there is silence in a video then it skips to where there is not silence? Also what threshold for silence do you think would be best?

[truthinhoney]

I sincerely apologize for the delayed replies.

No I know what skipping silence does, I am saying how does it work? When there is silence in a video then it skips to where there is not silence? Also what threshold for silence do you think would be best?

Unfortunately I'm code illiterate and I cannot understand how the SilenceSkippingAudioProcessor.java works exactly. I would put in the work myself and make a pull/merge request if I knew how anything about programming.

I was hoping that adding the Skip Silence feature would be as simple as implementing the code above since it is licensed as Apache V2, but I acknowledge that it is probably much more complicated than that. It might need to be entirely rewritten in another programming language for it to be compatible with FreeTube :crying_cat_face:

I strongly recommend downloading NewPipe and enabling Silence Skipping to try it out yourself if you have access to an Android device.

The Silence Skipping simply jumps over any portion of the media that does not have audio. I'm not 100% certain if it only skips over muted segments or if it compares the audio volume levels relative to the rest of the media. Having a threshold would be a smart way to handle it.

I'm not interested in adding any performance regression to the FreeTube player. Low-powered Android devices seem to handle Silence Skipping easily, it should not require too much additional hardware power to calculate audio volume levels.

I recognize that this feature is not exactly popular, otherwise it would be commonly available in media players.

[Rabcor]

What would skipping silence be used for ?

some audio tracks have huge dead zones where you have several minutes of silence, although this mostly only happens in really old music uploads.

[x-N0]

I'd love to see this!

[x-N0]

What would skipping silence be used for ?

See videos faster (like tutorials) without increasing the actual speed of the video, so you understand the words said but without having to wait for silences or long pauses.

To me when listening podcasts having so little time available for myself, makes a huge difference. We have a similar feature in the sibling app called NewPipe.

[toas-koas]

Is there any update?

[truthinhoney]

I accidentally closed this issue thread.

@PrestonN Can you please move this back to the To Assign in Feature Requests?

[toas-koas]

Is there any news?

[github-actions[bot]]

This issue is stale because it has been open 28 days with no activity. Remove stale label or comment or this will be closed in 7 days.

[yshalsager]

I came across https://github.com/vantezzen/skip-silence today, and thought something similar can be implemented in FreeTube.

[typedashutosh]

I came across https://github.com/vantezzen/skip-silence today, and thought something similar can be implemented in FreeTube.

I used it but it doesnot sync audio and video in free version, rather use Jump cutter Jump cutter

It would be very nice if skipping silence would be natively available in FreeTube. I am just looking for windows version of TubeMate

[Geniusssmit]

[ZinRicky]

I hope this will be implemented someday

[andraz]

This is an impressive functionality of the NewPipe application for common issues such as: excessive silences, drawn-out transitions, and unnecessary pauses often found in various types of content.

By intelligently reducing the duration of these problematic segments, the tool enables viewers to enjoy media at a more manageable pace, without needing to resort to extreme speed adjustments. For instance, lengthy and sluggish discussions, which may previously have required a 280% increase in playback speed to remain tolerable, can now be viewed comfortably at a reduced rate of 225%, thereby maintaining audio clarity while minimizing viewer frustration.

https://www.reddit.com/r/NewPipe/comments/r4selj/completely_addicted_to_fast_forward_on_silence/

I guess we'll need to install Android emulator to get decent features available on PC.

[AvAkanksh]

when can we expect his feature to be live ? many of them are eagerly waiting for the feature to be up.

[ahaone]

The implementation of the feature here works well. https://github.com/libre-tube/LibreTube/issues/2731

[absidue]

Just to be clear NewPipe and LibreTube use a video player (Google's ExoPlayer) that has that functionality built-in, so all those apps have to do to is turn that functionality on and off in that video player. So linking to those two apps doesn't help FreeTube, as it just tells us how to turn it on and off in that specific Android video player.

FreeTube would have to implement that functionality from scratch, as it is neither built into the video player's in browser's nor is it built into video.js, which is the framework that FreeTube uses that wraps around the browser's built-in player.

[andraz]

Great point, now we know what we're looking for. I did dig out the code below which is performing the logic.

https://github.com/google/ExoPlayer/blob/release-v2/library/core/src/main/java/com/google/android/exoplayer2/audio/SilenceSkippingAudioProcessor.java

Next step from here is to convert it into a more modern programming language, so we can actually use it in a normal application.

I used AI to generate the explanation how it works, and a reimplementation for JavaScript. Hope materials below can be used as a starting point to dig deeper.

---

Analysis of SilenceSkippingAudioProcessor

This Java code defines an AudioProcessor called SilenceSkippingAudioProcessor. Its purpose is to remove silent sections from an audio stream, reducing the data size and potentially improving playback efficiency.

How it works

Configuration

The processor is configured with three parameters:

• minimumSilenceDurationUs: The minimum duration (in microseconds) of silence required to be considered for removal.
• paddingSilenceUs: The duration of silence (in microseconds) to add after a non-silent section. This prevents abrupt transitions.
• silenceThresholdLevel: The absolute value of a PCM sample below which it is considered silent.

State Management

The processor maintains its state based on the incoming audio data:

• STATE_NOISY: The current audio is not silent.
• STATE_MAYBE_SILENT: The current audio might be silent, but more data is needed to confirm.
• STATE_SILENT: The current audio is confirmed to be silent.

Processing Logic

Input Processing

The processor reads the input audio data in chunks.

Depending on the current state, it performs different actions:

• STATE_NOISY:

  • It searches for the beginning of a potential silent section.
  • If found, it switches to STATE_MAYBE_SILENT.
  • Otherwise, it outputs the non-silent data.

• STATE_MAYBE_SILENT:

  • It accumulates data in a maybeSilenceBuffer until it reaches the minimumSilenceDurationUs.
  • If noise is encountered before the buffer is full, it outputs the buffer content and switches back to STATE_NOISY.
  • If the buffer is filled with silence, it switches to STATE_SILENT.

• STATE_SILENT:

  • It skips over silent data, keeping track of the skipped frames.
  • If noise is encountered, it outputs the paddingBuffer (containing silence from before the noisy section) and switches back to STATE_NOISY.

Output

The processor outputs audio data only when it encounters non-silent sections or when transitioning from silence to noise (padding). It keeps track of the total number of skipped frames.

Additional Notes

• The processor only handles 16-bit PCM audio format.
• It can be enabled/disabled dynamically.
• Flushing the processor resets its state and internal buffers.

Reimplementation in JavaScript

While a direct translation of the Java code to JavaScript is possible, it's more practical to adapt the logic to utilize JavaScript's strengths. Here's a possible implementation using Web Audio API:

class SilenceSkippingAudioProcessor extends AudioWorkletProcessor {
  constructor(options) {
    super();
    this.minimumSilenceDuration = options.minimumSilenceDuration;
    this.paddingSilenceDuration = options.paddingSilenceDuration;
    this.silenceThreshold = options.silenceThreshold;
    this.state = 'noisy';
    this.maybeSilenceBuffer = [];
    this.paddingBuffer = [];
    this.skippedFrames = 0;
  }

  process(inputs, outputs) {
    const input = inputs[0];
    const output = outputs[0];

    for (let channel = 0; channel < input.length; channel++) {
      const samples = input[channel];
      for (let i = 0; i < samples.length; i++) {
        this.processSample(samples[i]);
      }
    }

    // Output padding if transitioning from silent to noisy
    if (this.state === 'noisy' && this.paddingBuffer.length > 0) {
      output[0].set(this.paddingBuffer);
      this.paddingBuffer = [];
    }

    return true;
  }

  processSample(sample) {
    switch (this.state) {
      case 'noisy':
        if (Math.abs(sample) <= this.silenceThreshold) {
          this.maybeSilenceBuffer.push(sample);
          if (this.maybeSilenceBuffer.length >= this.minimumSilenceDuration) {
            this.state = 'silent';
          }
        } else {
          this.outputMaybeSilenceBuffer();
        }
        break;
      case 'silent':
        if (Math.abs(sample) > this.silenceThreshold) {
          this.paddingBuffer = this.maybeSilenceBuffer.slice(-this.paddingSilenceDuration);
          this.state = 'noisy';
          this.outputMaybeSilenceBuffer();
        } else {
          this.skippedFrames++;
        }
        break;
    }
  }

  outputMaybeSilenceBuffer() {
    if (this.maybeSilenceBuffer.length > 0) {
      // Output buffered data with padding
      const output = this.maybeSilenceBuffer.slice(
        0,
        -this.paddingSilenceDuration
      );
      this.port.postMessage(output);
      this.maybeSilenceBuffer = [];
    }
  }
}

registerProcessor('silence-skipping-audio-processor', SilenceSkippingAudioProcessor);

This implementation utilizes the Web Audio API's AudioWorkletProcessor interface to process audio samples directly. It maintains the same state logic and processing steps as the Java code, but uses JavaScript arrays and functions for buffer management and sample manipulation.

Note that this is a simplified example and may require further adjustments depending on the specific use case and integration with the Web Audio API.

[lonix1]

My dupe issue was closed, so adding my use case here:

Now that we have the playlists feature (BTW: thanks!), I often find that there is excessive silence between at the end if videos. It would be nice to have a "skip silence" toggle button in the playlist panel (loop, shuffle, reverse, SKIP, previous, next, pause).

The player seems to buffer some time ahead, so in theory it would be able to detect that

• it is near the end (e.g. < 20s) or at the beginning (==0s)
• the buffered portion that is still unplayed, is empty/silent

So unlike the use case above, this use case is for silence just at the end of videos. The silence anywhere else in the video is intentional and should not be skipped.