Self implementation - some confusion

[garrynewman]

Hey guys, I've been implementing Steam Audio and I've got a few high level questions.

Optimal Design

While the documentation does a good job explaining the components and what they're used for, it's not totally clear how they should be implemented. What I mean by that is, in terms of the graph, the order in which stuff is called.

So my impression was

( each sample [ samples > directeffects > reflection > ambisonicdecode ] )-> [output]

But the documentation seems to imply that reflection per source is prohibitively expensive - especially at realtime. So it would be better to do this:

( each sample [ samples > directeffects-> binaural ] ) > reflection > ambisonicdecode > [output]

But then I'm a bit confused, because reflection can only take a mono input, so does this mean that when doing listener based reflection it's always going to be mono? Or would you do it individually for each channel?

Pathing

This is for baked only, right? Or can runtime drop probes to populate/refresh in realtime?

Reverb

Reverb plays longer than the sample, and I assume you check the return of iplReflectionEffectApply to see if it's finished and then you can end that source. My assumption is that if you feed it silent buffers after the sample has finished, eventually it'll return IPL_AUDIOEFFECTSTATE_TAILCOMPLETE to indicate that it's no longer playing any reverb.

This doesn't seem to be the case for me, sometimes it doesn't return IPL_AUDIOEFFECTSTATE_TAILCOMPLETE at all. What could I be doing wrong?

Custom Tracer

I set up a custom tracer, and it's working great for occlusion, but I'm having problems with reflection. When I try to use IPL_REFLECTIONEFFECTTYPE_PARAMETRIC , it works, but whenever I use IPL_REFLECTIONEFFECTTYPE_CONVOLUTION , in iplReflectionEffectApply, it outputs a silent buffer.

Is this a limitation of the custom scene tracer, or should it work? Is there any way to debug why it isn't working? There's nothing from the debug print callback, the rays are definitely running and they're correct. I've been changing vars and moving stuff around for like 6 hours, so I thought I'd better make sure that it's meant to work before I did anything else.

[lakulish]

@garrynewman These are all great questions! Responses below.

Optimal Design

Applying reflections to individual sources isn't prohibitively expensive, but it does incur a non-trivial CPU cost, so per-source reflections should be used with careful consideration. That said, note that the way the DSP effects should be connected roughly mirrors the separation between direct sound propagation (sound traveling in a straight line from source to listener), and indirect sound propagation (reflections). So the way we do it in our plugins is:

             [direct effect] ------- [binaural/panning effect] 
           /                                                   \
[input] --                                                      + -- [output]
           \                                                   /
             [reflections effect] -- [ambisonic decode effect]

Here, both the direct effect and reflections effects take mono input. The direct effect generates mono output, which is spatialized by either the binaural or panning effects (depending on speaker configuration). The reflections effect generates Ambisonic output, which is spatialized by the Ambisonic decode effect. Both these spatialized outputs are then mixed to produce the final output.

Pathing

Yes, pathing is baked only at the moment.

Reverb

iplReflectionEffectApply always returns IPL_AUDIOEFFECTSTATE_TAILCOMPLETE because it assumes that there might be more input, even after you pass it a silent buffer (for example, you may be playing a speech clip with a brief pause). To address your use case, we would have to add new API functionality to explicitly indicate that the input has ended, and to fetch the reverb tail.

Custom Tracer

No, the custom ray tracer should work identically regardless of which reflection effect type you're using. Can you send a minimal repro program/project that we can use to diagnose the issue? Thanks!

[garrynewman]

Thank you, this all helped a lot (not as much as having the source though!).

I fixed the custom tracer not working. The problem was that it was sending infinity for the trace length, and I was trying to convert that to inches!