[C Api] v4.1.3 ReflectionEffect cannot run correctly and the output is silent (test case)

[comeonlby]

According to the documentation, ReflectionEffect is called，but it cannot run correctly and the output result is silent. (Tested with Steam Audio 4.1.3, MacOS 10.15.7 Intel, Xcode 11.3.1) Is this a bug or am I using the API incorrectly?

Click to expand

#include <algorithm>
#include <fstream>
#include <iterator>
#include <vector>

#include "phonon.h"
#include "WavFile.h"


int main(int argc, char** argv)
{
    argv[1] = (char *)"speech1.wav";
    argv[2] = (char *)"out_speech1_Reflection_Effect.wav";
     
    WavInFile *fileReader = new WavInFile(argv[1]);
    if(fileReader ==NULL){
        printf("in file is null \n");
    }
    int sampleRate = fileReader->getSampleRate();
    int numBits = fileReader->getNumBits();
    int channels = fileReader->getNumChannels();
    WavOutFile *fileWriter = new WavOutFile(argv[2], sampleRate, numBits, 4);
    int framesize = 1024;
    float *fInBuffer = (float *)malloc(sizeof(float)*(framesize * channels));
    float *fOutBuffer = (float *)malloc(sizeof(float)*(framesize * 4));
    IPLerror error;
    
    // Context
    IPLContextSettings contextSettings{};
    contextSettings.version = STEAMAUDIO_VERSION;

    IPLContext context{};
    error = iplContextCreate(&contextSettings, &context);
    if(error!=IPL_STATUS_SUCCESS)
    {
        printf("error=%d\n",error);
        return 0;
    }

    // Audio
    IPLAudioSettings audioSettings{};
    audioSettings.samplingRate = sampleRate;
    audioSettings.frameSize = 1024; // the size of audio buffers we intend to process
    
    // Scene
    IPLSceneSettings sceneSettings{};
    sceneSettings.type = IPL_SCENETYPE_DEFAULT;      //IPL_SCENETYPE_EMBREE

    IPLScene scene = nullptr;
    error = iplSceneCreate(context, &sceneSettings, &scene);
    if(error!=IPL_STATUS_SUCCESS)
    {
        printf("error=%d\n",error);
        return 0;
    }
    
    // four vertices of a unit square in the x-y plane
    IPLVector3 vertices[4] = {
        {0.0f, 0.0f, 0.0f},
        {1.0f, 0.0f, 0.0f},
        {1.0f, 1.0f, 0.0f},
        {0.0f, 1.0f, 0.0f}
    };

    // triangle indices use counter-clockwise winding order
    IPLTriangle triangles[2] = {
        {0, 1, 2},
        {0, 2, 3}
    };

    IPLMaterial materials[1] = {
        { {0.1f, 0.1f, 0.1f}, 0.5f, {0.2f, 0.2f, 0.2f} }
    };

    // both triangles use the same material
    IPLint32 materialIndices[2] = {0, 0};

    IPLStaticMeshSettings staticMeshSettings{};
    staticMeshSettings.numVertices = 4;
    staticMeshSettings.numTriangles = 2;
    staticMeshSettings.numMaterials = 1;
    staticMeshSettings.vertices = vertices;
    staticMeshSettings.triangles = triangles;
    staticMeshSettings.materialIndices = materialIndices;
    staticMeshSettings.materials = materials;

    IPLStaticMesh staticMesh = nullptr;
    error = iplStaticMeshCreate(scene, &staticMeshSettings, &staticMesh);
    if(error!=IPL_STATUS_SUCCESS)
    {
        printf("error=%d\n",error);
        return 0;
    }
    
    iplStaticMeshAdd(staticMesh, scene);
    iplSceneCommit(scene);
    
    // Simulation
    IPLSimulationSettings simulationSettings{};
    simulationSettings.flags = IPL_SIMULATIONFLAGS_REFLECTIONS; // this enables reflection simulation
    simulationSettings.sceneType = IPL_SCENETYPE_DEFAULT;
    simulationSettings.reflectionType = IPL_REFLECTIONEFFECTTYPE_CONVOLUTION; // see below
    simulationSettings.maxNumRays = 4096;
    simulationSettings.numDiffuseSamples = 32;
    simulationSettings.maxDuration = 2.0f;
    simulationSettings.maxOrder = 1;
    simulationSettings.maxNumSources = 8;
    simulationSettings.numThreads = 1;
    simulationSettings.samplingRate = audioSettings.samplingRate;
    simulationSettings.frameSize = audioSettings.frameSize;
    
    IPLSimulator simulator = nullptr;
    error = iplSimulatorCreate(context, &simulationSettings, &simulator);
    if(error!=IPL_STATUS_SUCCESS)
    {
        printf("error=%d\n",error);
        return 0;
    }
    
    iplSimulatorSetScene(simulator, scene);
    iplSimulatorCommit(simulator);
    
    // Source
    IPLSourceSettings sourceSettings{};
    sourceSettings.flags = IPL_SIMULATIONFLAGS_REFLECTIONS; // this enables reflections simulator for this source

    IPLSource source = nullptr;
    error = iplSourceCreate(simulator, &sourceSettings, &source);
    if(error!=IPL_STATUS_SUCCESS)
    {
        printf("error=%d\n",error);
        return 0;
    }
    
    iplSourceAdd(source, simulator);
    iplSimulatorCommit(simulator);
    
    // SimulationInputs
    IPLCoordinateSpace3 sourceCoordinates{.right  = IPLVector3{1, 0, 0},
                                          .up     = IPLVector3{0, 1, 0},
                                          .ahead  = IPLVector3{0, 0, 1},
                                          .origin = IPLVector3{1, 1, 1}};      // the world-space position and orientation of the source
    IPLCoordinateSpace3 listenerCoordinates{.right  = IPLVector3{1, 0, 0},
                                            .up     = IPLVector3{0, 1, 0},
                                            .ahead  = IPLVector3{0, 0, -1},
                                            .origin = IPLVector3{0, 0, 0}};     // the world-space position and orientation of the listener

    IPLSimulationInputs inputs{};
    inputs.flags = IPL_SIMULATIONFLAGS_REFLECTIONS;
    inputs.source = sourceCoordinates;

    iplSourceSetInputs(source, IPL_SIMULATIONFLAGS_REFLECTIONS, &inputs);

    IPLSimulationSharedInputs sharedInputs{};
    sharedInputs.listener = listenerCoordinates;
    sharedInputs.numRays = 4096;
    sharedInputs.numBounces = 16;
    sharedInputs.duration = 2.0f;
    sharedInputs.order = 1;
    sharedInputs.irradianceMinDistance = 1.0f;

    iplSimulatorSetSharedInputs(simulator, IPL_SIMULATIONFLAGS_REFLECTIONS, &sharedInputs);

    // typically run in a separate thread
    iplSimulatorRunReflections(simulator);
    
    // ReflectionEffect
    IPLReflectionEffectSettings ReffectSettings;
    ReffectSettings.type = IPL_REFLECTIONEFFECTTYPE_CONVOLUTION;
    ReffectSettings.irSize = 96000; // 2.0f (IR duration) * 48000 (sampling rate)
    ReffectSettings.numChannels = 4;
    
    IPLReflectionEffect Reffect = nullptr;
    error = iplReflectionEffectCreate(context, &audioSettings, &ReffectSettings, &Reffect);
    if(error!=IPL_STATUS_SUCCESS)
    {
        printf("error=%d\n",error);
        return 0;
    }

    // ReflectionMixer
    IPLReflectionMixer Mixer = nullptr;
    error = iplReflectionMixerCreate(context, &audioSettings, &ReffectSettings, &Mixer);
    if(error!=IPL_STATUS_SUCCESS)
    {
        printf("error=%d\n",error);
        return 0;
    }
    
    IPLAudioBuffer inBuffer;
    iplAudioBufferAllocate(context, channels, audioSettings.frameSize, &inBuffer);

    IPLAudioBuffer outBuffer;
    iplAudioBufferAllocate(context, 4, audioSettings.frameSize, &outBuffer);


    // Process
    int frame_count = 0;
    while (!fileReader->eof())
    {
        frame_count ++;
        printf("frame_count=%d\n",frame_count);
        int realSampleCnt = fileReader->read(fInBuffer, framesize * channels);
        int sample_per_channel = realSampleCnt/channels;
        
        iplAudioBufferDeinterleave(context, fInBuffer, &inBuffer);
        
        // typically run in the main update thread
        IPLSimulationOutputs outputs{};
        iplSourceGetOutputs(source, IPL_SIMULATIONFLAGS_REFLECTIONS, &outputs);
        
        IPLReflectionEffectParams Rparams = outputs.reflections; // this can be passed to a reflection effect (see below)
        Rparams.type =  simulationSettings.reflectionType;
        Rparams.ir = outputs.reflections.ir;
        Rparams.irSize = 96000;
        Rparams.numChannels = 4;
        
        iplReflectionEffectApply(Reffect, &Rparams, &inBuffer, &outBuffer, nullptr);
        
        iplReflectionMixerApply(Mixer, &Rparams, &outBuffer);
        
        iplAudioBufferInterleave(context, &outBuffer, fOutBuffer);
    
        fileWriter->write(fOutBuffer, sample_per_channel*4);
    }

    // Release
    iplAudioBufferFree(context, &outBuffer);
    iplReflectionEffectRelease(&Reffect);
    iplReflectionMixerRelease(&Mixer);
    iplContextRelease(&context);
    iplSceneRelease(&scene);
    iplSourceRelease(&source);
    iplSimulatorRelease(&simulator);

    delete fileReader;
    delete fileWriter;
    
    return 0;
}

[lakulish]

From what I can tell, this code tries to get mixed reflections from an IPLReflectionMixer object, but doesn't send audio to it from iplReflectionEffectApply. Try one of the following:

• Pass Mixer when calling iplReflectionEffectApply: iplReflectionEffectApply(Reffect, &Rparams, &inBuffer, &outBuffer, &Mixer);, or
• Comment out the call to iplReflectionMixerApply. This will directly render the output of the IPLReflectionEffect.

Let me know if neither of these resolves the issue.

[comeonlby]

From what I can tell, this code tries to get mixed reflections from an IPLReflectionMixer object, but doesn't send audio to it from iplReflectionEffectApply. Try one of the following:

• Pass Mixer when calling iplReflectionEffectApply: iplReflectionEffectApply(Reffect, &Rparams, &inBuffer, &outBuffer, &Mixer);, or
• Comment out the call to iplReflectionMixerApply. This will directly render the output of the IPLReflectionEffect.

Let me know if neither of these resolves the issue.

Thank you for your patient answer. I tried both of the methods you provided, but the final output is still silent.I suspect there is a problem with the generation of IPLReflectionEffectIR, but I don't have any log prompts.

[lakulish]

I took a closer look at the code, and it looks like the impulse response is silent because the listener is at one of the vertices of the geometry (0, 0, 0). Try moving the listener slightly away from the geometry (e.g. by adding an offset in the z-axis). That, in addition to one of the fixes I suggested above, should hopefully resolve the issue. Let us know if it doesn't.

[comeonlby]

I took a closer look at the code, and it looks like the impulse response is silent because the listener is at one of the vertices of the geometry (0, 0, 0). Try moving the listener slightly away from the geometry (e.g. by adding an offset in the z-axis). That, in addition to one of the fixes I suggested above, should hopefully resolve the issue. Let us know if it doesn't.

You're right. I tried to move the listener slightly away from the geometry and it worked. Thank you again for your patient answer, which is of great help to me. Best wishes.