Add interactive music support

[reduz]

trafficstars

This PR adds 3 types of audio streams used for interactive music support.

• AudioStreamInteractive: Allows setting several sub-streams and edit the transition conditions and options between them.
• AudioStreamPlaylist: Allows sequential or shuffled playback of a list of streams.
• AudioStreamSynchronized: Allows synchronous playback of several streams, the volume of each can be controlled.

These three stream types can be combined to create complex, layered interactive music and transitions between them, similar to software such as WWise, FMOD or Elias.

Screens:

Preview video (older version): https://www.youtube.com/watch?v=ftYVhMpBk4Y

They are all tiny classes that can be mixed and matched, in true Godot style. This supersedes #62896 and #32769.

Bonus: Added an extra PROPERTY_HINT_FLAG_LIST for editing flags as a list. This is useful when you have far too many flags. This is needed here for editing transitions without entirely bloating the inspector.

[Mickeon]

A new PROPERTY_HINT_FLAG_LIST? Where is it used on in here?

[reduz]

@Mickeon Its used for the transitions

[fire-forge]

After this is merged, it would be nice to add an @export_flag_list annotation to GDScript (and a C# equivalent) for exporting flags with the new PROPERTY_HINT_FLAG_LIST.

[MJacred]

First few findings (will test more)

Case 1
reproduction: add AudioStreamOggVorbis stream to AudioStreamPlayer or its stream playlist (did not test other stream types) what happens: spams modules/vorbis/audio_stream_ogg_vorbis.cpp:474 - Condition "packet_sequence.is_null()" is true. Returning: 0 to console

Case 2
Open an ogg file in advanced import window: eats 100% CPU (i.e. 1 core), normally ~8%
Set beat count and bar beats to 4, then play preview: after playing the stream to the fake ending, CPU jumps over 200%
Both high consumptions mentioned before are only released on quit to project list or quit app
maybe related warning: modules/vorbis/audio_stream_ogg_vorbis.cpp:324 - Burning negative samples doesn't make sense. Check seek algorithm.
may also make godot unresponsive (killing with SIGTERM signal with terminal still works)
EDIT: upon re-testing, the 100% cpu by just opening advanced import window is not reproducible, but the latter high consumption with incorrect beat settings persists.

Proposals

• Personally, I find it odd to enter the total beat count and bar beats instead of entering time signature, e.g. 4/4
• rename beat count to total beat count to better differentiate from bar beats, or add fitting Editor description Total beat count within the audio stream.

[MJacred]

Case 3
Create a new AudioStreamPlayer and set stream type to e.g. Playlist, then add your first stream and drag and drop in an ogg-file. First preview in "Stream" is black (which is fine) and once it's done scanning the stream, it shows a squished preview Manual fix: select a different node in the scene and then back to the AudioStreamPlayer

Case 4
Deleting a stream/clip in Playlist, Interactive, … does not clean up the stream/clips list: not-deleted.webm

Proposal 1
It would be nice to be able to change the stream position in the Inspector (i.e. seeking), makes testing faster/easier For reference, this is how I implemented it in my adaptive music addon (see at top the jump button):

Proposal 2
Add option to copy Audio Streams in Playlist & Co., in case you want to switch from Synchronized to Interactive, otherwise you have to re-drag & drop all files. And then paste on new stream type

NOTE

• while playing preview in Inspector, manipulating volume db, pitch scale, playing, stream paused, mix target, bus and (un)plugging headphones worked without issue

[MJacred]

Case 5 Audio pop -> created Issue: https://github.com/godotengine/godot/issues/64775

Case 6
Ok, I get the modules/vorbis/audio_stream_ogg_vorbis.cpp:324 - Burning negative samples doesn't make sense. Check seek algorithm warning sometimes, when seeking in the advanced importer, while the stream is playing. (NOTE: BPM was disabled, but loop point was defined.)

Proposal 3 In Interactive, you can define a clip name, which is auto-generated on drop (if no clip name defined). But it is not updated in case it could; reproduction

1. drag&drop file my_test_song.ogg and results in clip name My Test Song
2. don't change name -> drag&drop file rock_song.ogg -> clip name stays unchanged

So in case name is not adjusted by user, re-generate clip name using audio file name.

[MJacred]

Case 7 (might also be a security feature to avoid breaking everything.)
Create Interactive AudioStreamPlayer, then drag&drop in clips.
Rename a Clip.
Result: Clip name references in Transitions and "initial clip" and "switch to" stay outdated.
Only workaround that works: re-opening the scene.

will test more another day…

Until then, what does AUTO_ADVANCE_RETURN_TO_HOLD do?

[reduz]

@MJacred thanks so much for testing! Transitions have the hold option so they remember which clip they come from. Return to hold just returns to that clip. This is useful for when you have the main theme playing and you want to transition to an incidental music clip, battle clip, etc. then go back to wherever you were before. The PreviousPosition option also helps to return to the previous saved state in case you want to continue from where it was (very common for the main theme).

[MJacred]

Glad to be of help and thanks for the explanation! Will check it out another day and do more testing :sleepy: :zzz:

Ah, one last thing today: Another feature I really missed in the Inspector: the current TIME, i.e. the progress in seconds:milliseconds of the currently playing clip(s). A must have, especially if you add a seek helper-field (mentioned in https://github.com/godotengine/godot/pull/64488#issuecomment-1216827394 under proposal 1)

---

EDIT2
NOTE: testing will continue on August, 23nd (did lots of hterrain testing today…)

[ghost]

Will it be available on 3.5?

[Calinou]

Will it be available on 3.5?

No, as this PR relies on backwards-incompatible changes to AudioServer.

[IntangibleMatter]

Is there any way to build this for Windows? I've been trying since the pull request came out and it consistently returns Error 1 in the last stage of compilation.

[fire-forge]

@IntangibleMatter This pull request seems to be a work in progress right now. If you look at the checks it's failing for almost all platforms.

[TokageItLab]

@reduz I provide some feedback.

---

Is it possible to set a property like Global BPM in AudioStreamInteractive and set a switching point for such like every bar or every 4 bars?

Especially in music with a clear pattern of beats, there are several points suitable for switching, like in a DJ Mix. It is better to keep the current Stream until the point after the switching command, and switch when the point is reached.

---

~~Can you implement something like AudioStreamRoundRobin as a variation of AudioStreamPlaylist?~~

~~I wrote a Proposal on this in https://github.com/godotengine/godot-proposals/issues/5231. If we want to randomize the playback of footsteps with patterns, AudioStreamPlaylist will result in a long AudioStream. To make it more flexible and versatile, it would be excellent if you could integrate that proposal into this PR if possible.~~

There is already exist as AudioStreamRondomizer.

[MJacred]

Is it possible to set a property like Global BPM in AudioStreamInteractive and set a switching point for such like every bar or every 4 bars?

The time signature / bpm can be different per audio file, so having one for all of them would be detrimental in other cases…
See this wonderful lecture by Chance Thomas: https://youtu.be/q4CYUfgRdos?t=1680
Having the transition/swap on the next smallest common denominator bar would be better

so this would be a new
Proposal 4: for Transitions, add

• From Time: next common bar
• To Time: Same Position (as we wait for next common bar, this should be fine)

[MJacred]

proposal 5: mass loader
in the addon (it's coming), there's a mass loader. In this example the there are only 3 streams in that track, but we have other tracks with 10 streams/files. and so on. becomes tiresome. If you want some inspiration:

https://user-images.githubusercontent.com/6639237/186237772-36389804-c700-4141-b959-9057254c20f7.mp4

[MJacred]

case 8: Auto Advance enabled ignores looping -> might have found root cause, see code review
After clip 0 finishes, it auto-advances to clip 1, but it does not jump to clip 1 loop point, it jumps to START.
Therefore, Same Position should mean loop point, if stream changes clip on file end. An alternative would be to add new To Time option Loop Point. The latter would be confusing though, because we already have a To Time option Start

[IntangibleMatter]

Is it possible to set a property like Global BPM in AudioStreamInteractive and set a switching point for such like every bar or every 4 bars?

I honestly think what a good approach would be (that I've been hesitating to suggest because it would require changing some of the fundamental code of this pr, as far as I can tell) would be to be able to set what multiple of beats/bars is transitions on. I.E. a song could be able to transition on every OTHER bar because it would better match the pattern of the song.

[MJacred]

case 9: Hold Previous reduces volume dB

1. click Playing in inspector to play interactive stream player -> plays clip 0
2. switch to clip 1 in inspector

click Hold Previous (on transition for clip 0 to clip 1) at any time BEFORE starting to play or BEFORE manually switching to clip 1

This happens only on FADE_IN and FADE_CROSS

What happens: stream continues playing and reduces volume dB to reeeeally quiet. no error/warning

BUT! If you do a step 3 and switch back from clip 1 to 0, the volume dB is reset. This does only work if step 3 is a switch back to the clip where volume dB was working correctly. Switching to any other clip and then sometime back to clip 0 (where volume dB was working correctly at play start), the volume dB won't be fixed. EXCEPT you let it play on, wait for its auto-advance to trigger to another clip, then it suddenly plays volume dB correctly again. Confusing, I know.
So in short: if your clip switch mutes volume, fix it either by a) going back to previous clip where volume was fine or b) play any other clip, live with the mute, then auto-advance to another clip and volume dB is back…

[MJacred]

case 10: ReturnToHold only works AFTER transitioning at least once to a clip with no ReturnToHold

1. Play clip 0
2. let it (auto-)advance to clip 2
3. let it (auto-)advance to clip 1
4. jumps back to held clip 0 -> but this sounds like a bug :bug: , because before clip 1, clip 2 was playing… right?

this way does NOT work, see code highlighted below in my review:

1. Play clip 0
2. let it (auto-)advance to clip 1
3. will never jump back to held clip 0

Might have found root cause, see code review

But it works, if you immediately manually switch from clip 0 to 1.

I foresee loads of misunderstandings concerning ReturnToHold and Hold Previous

Will continue testing in the coming days…

still leftover

• fading (mode, beats)
• filler
• test playing utilities with wav and mp3
• more?

[MJacred]

case 11: does not advance on auto advance == Enabled in this process

1. play clip 0
2. manually switch to clip 1 right away
3. manually switch back to clip 0 right away
4. clip 0 will never auto-advance to clip 1, even though defined

same setup as case 10

Might have found root cause, see code review

[MJacred]

proposal 6: allow filler to play concurrently to next clip -> i.e. function as Stinger
Right now, if a filler is defined, it plays on a transition and then after it finishes, it plays the next clip.
If you allow to play the filler concurrently (boolean flag, so existing logic is not obstructed), instead as an inbetween, the filler can be used as a Stinger; often used to hide an ugly cross-fade. See: https://youtu.be/YFC8gV_bcwc?t=45

See code review for line/area where boolean would be read

[MJacred]

case 12: copy & make unique prints error

1. create setup above
2. create new AudioStreamPlayer2
3. copy properties from first player
4. paste into second player
5. on Interactive, make Unique

prints warning

modules/interactive_music/audio_stream_interactive.cpp:89 - Condition "p_clip < 0 || p_clip >= clip_count" is true.

[MJacred]

proposal 7: allow playing Stinger at any time during a clip
Building on proposal 6, it's sometimes necessary to play a Stinger when an in-game event happens.
Both can be worked around with using code and a second AudioStreamPlayer.

case 13: audio pops on Fade Modes Disabled and Out
might have same root cause as https://github.com/godotengine/godot/issues/64775

proposal 8: improve flexibility of cross-fade
My addon currently uses tweens for fade in, fade out and cross fade. By doing the right setup (see below), you can create your own cross-fade (EXPO + IN (for fade out) or OUT (for fade in) works really well, probably because exponential is the inverse of logarithmic).

Here, the cross-fade is symmetric, like it is implemented in your PR. But sometimes, you want the fade out to take longer than the fade in of your new clip/layer, depending on the BPM/mood/loudness/etc. Example: Fade out playing clip over 4 seconds, but fade in next clip over 2 seconds, potentially with a 1 or 2 second delay.

to summarise, Transition requires

• different fade length per clip
• delay for both fades

proposal 9: merge Synchronized and Interactive
allow multiple streams to play at the same time for additive adaptive music, see NieR: Automata: https://www.youtube.com/watch?v=jFEVGRJFcVg
You could do this with current Interactive, if you pre-merge the sound files in an external editor. But then you will need to create a lot more files -> because of the many combinations.

proposal 10: change fade beats into fade time (i.e. use float for seconds::milliseconds)
While using beats sounds nice to have, using float gives more control… Also used like this in Cubase: https://youtu.be/QV46xI7jHuk?list=PLNpTJ4IJdJqfio1aAwphtAIczWYs59op3&t=560

Closing words for today

I'll test wav and mp3 another day. Especially in regard to case 13. And Playlist…

Right now, proposal 8 is a showstopper for better cross-fade control in my addon. Proposal 10 is not a do or die, but still pretty strongly desired.

[IntangibleMatter]

Okay, so sadly I can't test this because it won't compile for Windows, but after reading the code the only feature I can really see needing any changing is adding an option in the TransitionFromTime enum in audio_stream_interactive.h to make it so you can define a custom multiple of beats/bars to change on. I.E. Maybe you only get a smooth change every other bar, or every 17 beats, or something else like that. Maybe it could expose a setting that's otherwise hidden like the emissionShape options in Particles2D?

[MJacred]

OK, today I did mostly code review. I found root causes of some cases. Though I didn't cover everything, there are some parts that leave me head scratching (my code editor is not fully setup, so references cannot be searched for; which does not help…). Last but not least, I did a cleanup of cases and proposals in previous comments with newer testing + code understanding.

EDIT: I also checked for case 9, but this one eludes me right now. It's probably something with speed or volume of fade. Maybe weird numbers that are not catched by any if-clause…

Proposal 6 can be worked around, I think, by using timer/tween logic. Not having proposal 8 would be a bit annoying (I'll update proposal 8 with realistic implementation notes next time).

This will have to wait until next time testing (either tomorrow or next week):

I'll test wav and mp3 another day. Especially in regard to case 13. And Playlist…

[MJacred]

@IntangibleMatter: if you do the fixes I mentioned in my 2 reviews regarding 64-bit shift, then it should compile for Windows

[MJacred]

Proposal 11: add AudioStreamPlaylist::get_list_stream_start_position
Right now, when hitting AudioStreamPlayer::play(500.0), this would play a list item at a specific point. But it's cumbersome if you want to start at a specific item/stream in the list.
The proposed function get_list_stream_start_position(7) would return e.g. 534.42 for the starting position of item 7

Requires almost the same calculation as used in AudioStreamPlaybackPlaylist::start

[MJacred]

Proposal 12: add filler to AudioStreamPlaylist
the equivalent of filler in AudioStreamInteractive. use case

• to play SFX for e.g. putting on a disc or cassette player
• play a silence track to add delay between playlist items
  • having a delay/wait parameter as an alternative would be, though

User can manually call AudioStreamPlaylist::set_transition_filler_clip to switch out the filler at runtime for variation of SFX

---

@reduz: does AudioStreamPlaybackPlaylist::get_playback_position() return the position within the currently playing stream or the total length of the playlist?

[reduz]

@MJacred Thanks hugely for all the effort and feedback! As I mentioned before, I don't really want to do something too complex because its likely beyond the point. This should hopefully cover most use cases. Instead I think that most of the components needed for someone who needs to create something more complex should be in place.

One thing I would prefer to not do is loop beginning positions > 0. Its difficult to get right for musicians since many times there are effects running and the loop sounds a bit weird. I think its better to just use intro clip and loop clip instead.

I will try to working on the fixes to the issues you found during sometime this week.