Split track selection to video and audio selection

[Arnei]

In the "Tracks" view, instead of de-/selecting the whole track, the video and audio streams can be de-/selected individually. At least one video stream must remain selected. All audio streams can be deselected. If the audio stream is marked as unavailable, it cannot be de-/selected.

Shows the waveform from the timeline for a graphical representation of the audio stream. However, there will only be one waveform generated for the timeline, so in case of additional tracks a placeholder image will be shown instead. Any suggestions/submissions for a better placeholder graphic are very welcome.

Resolves #1009. #1009 also mentions potential issues in the backend, while this PR only addresses the frontend. From my limited testing the backend seems fine when using the default community workflows (no errors, the correct streams get published). If there are any users around that are still using the track selection feature in the old editor, and that are aware of any backend issues, please do tell me about them.

Courtesy of the Opencast 2023 Crowdfunding.

[github-actions[bot]]

This pull request is deployed at test.editor.opencast.org/1230/2023-12-14_10-06-39/ . It might take a few minutes for it to become available.

[lkiesow]

There are definitely issues in the backend. But I also see usability issues in the frontend.

For example, what will be the result of a selection like this?

As a user, my assumption would be to get either a single video with mixed audio streams from the other tracks or three tracks, two with audio and one with video.

However, the muxing command being executed is:

[ffmpeg, -nostdin, -nostats, -i, ...fc9.mp4, -i, ...ad4.mp4, -shortest, -c, copy, -map, 0:v:0, -map, 1:a:0, ...b5-copy.mp4]

This completely discards one of the selected audio streams. As a user, I can just describe that as a bug.

And then, what would be the expected result of something like this?

Will all audio streams be mixed into a single audio stream and applied to A? Or will just C be transferred to A? As a user, you have no idea.

[Arnei]

My thoughts on how to go about the usability issues you highlighted:

1. Leave it as is, people will "somehow just know" how it works at their place. Arguably workable, but not a great solution.

2. Introduce restrictions on what combination of tracks can and cannot be selected. Would need to be configurable, because there will always be admins that want to allow combinations that others would restrict, like the ones you mentioned. But then we again run into the issue of having to "just know" what certain combinations will do.

3. Add more buttons/checkboxes/ui that describes what should happen with the tracks, e.g. Mux all remaining audio streams onto A/B/C or Keep audio on tracks, Allow tracks with only audio. Arguable the way to go even if it muddles up the UI somewhat, but also requires an editor workflow to support all those new flags.

[lkiesow]

The original goal of the editor was to be simple since it is meant for non-technical end-users. I would really like to keep that goal since we expose the editor to such users.

Therefor, a lot of implicit knowledge as a requirement isn't a great solution and neither is having a lot of boxes and buttons with technical terms.

As an alternative, we could introduce one simple way to handle the result. My suggestion would be: If you remove any audio stream, all other audio streams will be mixed and attached to all videos. This rule is simple and works with as many videos tracks as you want. It also works well with other parts of Opencast like the Paella Player, which will just play back a single stream.

Running with that idea, we would need a change in the backend workflow operation and a hint about how this works in the UI of this pull request. The hint should also include that the mixing will happen if a track is already missing an audio stream.

Furthermore, since users should be able to deal with individual audio streams, we also need to make sure the editor can playback all streams, and individual streams. A simple solution may be playback controls on the tracks page where you then disable the audio streams of the tracks were audio has been removed?

[github-actions[bot]]

This pull request has conflicts ☹ Please resolve those so we can review the pull request. Thanks.

[github-actions[bot]]

This pull request is deployed at test.editor.opencast.org/1230/2024-01-18_15-31-01/ . It might take a few minutes for it to become available.

[KatrinIhler]

Finally getting around to commenting on this... I will not chime in with the ongoing discussion, but just say that

At least one video stream must remain selected

is something I would like to see either being configurable in the backend, or simply not enforced (which I think you already knew I was gonna say ;)), because it doesn't take into account audio-only use cases. Consider the following:

1. An institution with custom workflows for publication that can handle audio-only input (either by rendering a static image into it, or by having a player that can handle audio-only), and
2. A recording where the video is either black or otherwise damaged or unusable, but the audio is fine, and I want to deselect the video and publish the audio only with my custom workflow, or
3. A recording that doesn't come with a video stream in the first place that I want to cut and publish.

Both of these cases should be possible with the new editor. It doesn't have to be the case out of the box, but I shouldn't need to patch the code to make this work, just do some configuration. So basically the same arguments I made here, except that for the new editor, making this configurable with an "allow audio only in editor" or something would actually be worth it.

[github-actions[bot]]

This pull request has conflicts ☹ Please resolve those so we can review the pull request. Thanks.

[luniki]

Lars sent me this "oneliner" to create a triple event:

curl -u admin:opencast "${SERVER}/ingest/addMediaPackage/fast" \
        -F 'flavor=presenter/source' \
        -F mediaUri=https://example.com/opencast/test-media/goat.mp4 \
        -F 'flavor=presentation/source' \
        -F mediaUri=https://example.com/opencast/test-media/goat.mp4 \
        -F 'flavor=experiment/source' \
        -F mediaUri=https://example.com/opencast/test-media/goat.mp4 \
        -F identifier=three \
        -F title="Three Video Streams" \
        -F creator="$(shuf -n 1 /usr/share/dict/words)"

[Arnei]

Results of todays discussion between Sascha, Katrin, Lars, Arne and Marcus:

• Should be able to select 0, 1 or more videos
• In the same vein, should be able to select 0, 1 or more audios
• At least one track (video or audio) needs to remain
• The same audio is applied to all videos: None, one or multiple muxed
  • Have a note in the UI that informs the user about muxing
• In case no video remains selected, the remaining audios are muxed into one audio track
• Default selection when opening up the tracks view: All videos are selected and only the first audio track is selected
  • ...why did we want this again?
• There should be a big confirm button
  • This idea here is to avoid a user making an accidental change to their event

Key-Image

[luniki]

I talked to @Arnei about this and created a first version of it. Here are some screenshots.

Initial state after opening the "Tracks" tab

After de-selecting a video track

After de-selecting all video tracks

After de-selecting all but one track

[luniki]

The "Confirm selection" buttons seems to be useless. Last we talked we defined:

There should be a big confirm button * This idea here is to avoid a user making an accidental change to their event

The initial state is that all tracks are selected. This can also be easily determined in the code. So if we never de-select any tracks, we don't change the event?!

[snoesberger]

The "Confirm selection" buttons seems to be useless. Last we talked we defined:

There should be a big confirm button

• This idea here is to avoid a user making an accidental change to their event

The initial state is that all tracks are selected. This can also be easily determined in the code. So if we never de-select any tracks, we don't change the event?!

IIRC, the idea was not to have a confirm button, but to have a checkbox to activate the track selection. So if a user wants to de-select tracks, they first have to activate this "modify track selection" checkbox. As long as the checkbox is inactive, video and audio selection is grayed-out, nothing can be changed and the track selection will be completely ignored. As soon as the checkbox is activated, the user will be able to de-select video and/or audio tracks, and Opencast will process video and audio accordingly (muxing audio, etc.) as soon as the user starts the workflow. @KatrinIhler and @lkiesow please correct me if I'm wrong.

[KatrinIhler]

Agreed with @snoesberger completely . The difference between an activated and deactivated track selection, if the user does not do anything else, is whether the audio tracks are kept as-is or are muxed.

So no to the confirm button (bc this doesn't persist this decision in a visible way later), yes to a checkbox or something to activate the whole selection thingy (as shown on the initial sketch).

[luniki]

I have pushed my changes to this PR to my own branch: https://github.com/luniki/opencast-editor/commits/tracks-select-audio/

[luniki]

Here are some screenshot of my current state of track selection.

Entering the track selection

Enabled track selection

De-selected all but one track

[luniki]

The checkbox labelled "Customize track selection" is a new part of the state of the editor which has to be transferred to the Opencast EditorService endpoint. At the moment there is an additional boolean key in the JSON object.

This requires changes in the EditorRestEndpointBase class (and the GSON classes).

Or do I get this wrong? How should that checkbox be encoded?

[luniki]

I pushed my changes to my forked branch of opencast-editor: https://github.com/luniki/opencast-editor/commits/tracks-select-audio/

In the backend I edited the EditingData class: https://gitlab.elan-ev.de/mlunzena/opencast-bern/-/commits/feature/track-selection

[github-actions[bot]]

This pull request is deployed at test.editor.opencast.org/1230/2024-06-06_06-58-25/ . It might take a few minutes for it to become available.

[github-actions[bot]]

This pull request is deployed at test.editor.opencast.org/1230/2024-06-06_11-07-39/ . It might take a few minutes for it to become available.

[snoesberger]

Thanks for the update. I have tested the latest changes and have the following comments:

• If there is a track without audio, this track is always selected and can't be deselected. This track should be deselected by default, because otherwise it is possible to publish this silent audio track without video, which makes no sense.

• If the video on the left has no audio, a placeholder image will be displayed instead of the generated waveform for the video on the right:

• Rather than having "No Audio" (which is styled differently to the rest), I'd prefer to have a simple line indicating that there is no audio.

[Arnei]

If there is a track without audio, this track is always selected and can't be deselected. This track should be deselected by default, because otherwise it is possible to publish this silent audio track without video, which makes no sense.

To phrase it another way:

• For tracks without audio, the audio should initially be deselected
• "No Audio" can later be selected (it is not locked into being deselected)
• It should not be possible to create selections that only contain "No audio"

Does that seem right to you?

[Arnei]

Rather than having "No Audio" (which is styled differently to the rest), I'd prefer to have a simple line indicating that there is no audio.

Sure, will do. If you have any other suggestions for the "No Audio" or "Placeholder" images, feel free to post them. I did not put too much effort into them initially and have no attachment to how they look.

[snoesberger]

Sure, will do. If you have any other suggestions for the "No Audio" or "Placeholder" images, feel free to post them. I did not put too much effort into them initially and have no attachment to how they look.

Or why don't we just remove the checkbox and image completely? If there is no audio track we shouldn't show one. Instead we could just show a "rectangle" as a placeholder:

This would also make your questions from the comment https://github.com/opencast/opencast-editor/pull/1230#issuecomment-2154255479 obsolete. What do you think?

[Arnei]

If the video on the left has no audio, a placeholder image will be displayed instead of the generated waveform for the video on the right:

Might this be because there actually is no generated waveform? Currently, a waveform will only be generated for the first track, even if that track has no audio (#1376 should fix that), which would be presentation in your image.

[Arnei]

Or why don't we just remove the checkbox and image completely? If there is no audio track we shouldn't show one. Instead we could just show a "rectangle" as a placeholder:

Might be because of my developer glasses, but to me that looks like the page is broken :D

[snoesberger]

If the video on the left has no audio, a placeholder image will be displayed instead of the generated waveform for the video on the right:

Might this be because there actually is no generated waveform? Currently, a waveform will only be generated for the first track, even if that track has no audio (#1376 should fix that), which would be presentation in your image.

But in my case there is already a waveform available, at least in the cutting section the waveform for the presenter is displayed.

[Arnei]

But in my case there is already a waveform available, at least in the cutting section the waveform for the presenter is displayed.

Okay, strange. I'll look into it, thanks for telling me.

[github-actions[bot]]

This pull request is deployed at test.editor.opencast.org/1230/2024-06-07_08-40-49/ . It might take a few minutes for it to become available.

[github-actions[bot]]

This pull request is deployed at test.editor.opencast.org/1230/2024-06-07_09-15-18/ . It might take a few minutes for it to become available.