producer audio level via OSC

[elanvrt]

It would be nice if CasparCG could send OSC messages showing the integrated rms audio level per frame for each audio channel in FFMPEG or video-in producers. Would enable to create a client that has audio level meters for each layer individually, instead of just being able to monitor the mixdown on the output.

[TomKaltz]

+1 !!

Thomas R. Kaltz III

(586) 214-7150 cell (866) 855-6350 fax

On Wed, Oct 23, 2013 at 10:20 AM, elanvrt [email protected] wrote:

It would be nice if CasparCG could send OSC messages showing the integrated rms audio level per frame for each audio channel in FFMPEG or video-in producers. Would enable to create a client with audio level meters.

— Reply to this email directly or view it on GitHubhttps://github.com/CasparCG/Server/issues/190 .

[coral]

Yeah this would be a good idea for making a simple audio monitor using OSC.

Same as #91 though.

[TomKaltz]

#202

[TomKaltz]

Please note that the recent feature addition is for the Channel mixer NOT for each producer. I think it would be too much of a performance hit to calculate levels for each consumer. Any thought?

[HellGore]

Yes, I agree that it might be too much. Every feature like this we add (features that have a performance impact regardless of producers/consumers used), takes a step back on the "you should not have to pay for what you don't use" scale (C++ philosophy which I think should apply to CasparCG as well).

Actually I see now that the issue description is about each layer, so the question is if the changes made:

• Adds wanted statistics although it says nothing about each layer, only what comes out of the mixer
• Only adds performance hit, without being wanted
• Will be useful only if each layer is presented as well/instead.

[tjenkinson]

As things like this might have a performance hit would it be possible to allow the user to enable and disable these features in the config? Maybe they could then also be turned on and off with an AMCP command?

[HellGore]

Yes it would be possible, actually the best approach might be to create a consumer (audio_level_reporter or similar) moving out this code from the audio_mixer (less intrusive and by nature optional), and also doing the volume-diag-graph there as well.

[ronag]

"Premature optimization is the root of all evil"¨

The amount of data for audio per frame is: ~1.536 MB/s. The amount of data for HD video per frame is: ~207.36 MB/s.

Just copying a single video frame to GPU mapped memory will probably take as much cpu as calculating dbFS for ~50x 8 channel audio streams.

The overhead for simple audio calculations is negligible. Especially if you let the compiler (or manually) SSE optimize it (VS2013 is pretty good at this).

Measure first, before thinking about overheads.

[HellGore]

Just talking about a possible overhead is not the same as performing a premature optimization.

It all depends on what one considers "negligible", every small feature we add to the core rendering pipeline adds something. It can be the difference between never experiencing frame drops to sporadically experiencing frame drops (we have already seen this with chroma key for example).

Of course small changes which is not of the type "do something with every pixel in a frame" or "do something with every audio sample in a frame" does not even qualify for discussion. But since this is a case of "do something with every audio sample in a frame" I think it warrants discussion.

Btw, it will be interesting to see what kind of overall performance improvement we will see with VS2013

[ronag]

Just thought I'd chime in since it seemed like you are already considering adding complexity just to make it optional.

To do the dbFS calculation for one frame on a single core on my laptop without SSE optimization takes less than 30 nanoseconds.

Decoding a qtrle HD frame takes ~5-15 milliseconds.

So the per producer overhead in that case would be less than 0.0002%

#include <windows.h>
#include <vector>
#include <stdlib.h>
#include <cstdint>
#include <iostream>
#include <algorithm>

#undef max
#undef min

int main()
{   
    LARGE_INTEGER start;
    LARGE_INTEGER end;
    LARGE_INTEGER freq;
    long long ticks = 0;

    const auto frame_rate = 25;
    const auto sample_rate = 48000;
    const auto num_channels = 8;
    const auto sample_per_frame = (48000 * num_channels) / frame_rate;

    std::vector<std::int32_t> result(sample_rate * num_channels);

    const auto k = 200;

    std::vector<int> dummy1(16*1000000);
    std::vector<int> dummy2(16*1000000);

    for (auto n = 0; n < k; ++n)
    {
        // This should be more than enough to flush cpu cache.
        std::generate(std::begin(result), std::end(result), rand);
        std::generate(std::begin(dummy1), std::end(dummy1), rand);
        std::copy(std::begin(dummy1), std::end(dummy1), std::begin(dummy2));

        QueryPerformanceCounter(&start);

        std::vector<std::int32_t> max(num_channels, std::numeric_limits<std::int32_t>::min());

        for (size_t n = 0; n < result.size(); n += num_channels)
            for (int ch = 0; ch < num_channels; ++ch)
                max[ch] = std::max(max[ch], std::abs(result[n + ch]));

        QueryPerformanceCounter(&end);

        ticks += end.LowPart - start.LowPart;
    }

    QueryPerformanceFrequency(&freq);

    auto elapsedTime = static_cast<double>(ticks)/static_cast<double>(freq.LowPart);

    std::cout << elapsedTime/(k * sample_per_frame) * 1000000000 << "ns/frame" << std::endl;

    system("PAUSE");
}

[HellGore]

I agree with this being negligible. Furthermore it is actually replacing the calculation:

auto max = boost::range::max_element(result);

Which you had there previously to calculate the volume-graph, so the added overhead is even less, since this iterating of all the samples is no longer done.

[elanvrt]

I am glad to see that the audio meter is actually in progress! It is indeed the intention of having a level meter per audio producer. Having one on the consumer output only, after the mixdown, is much less useful since one can use a hardware audio meter for that. If performance overhead is not negligible, it would indeed be useful to have an AMCP command to enable/disable the feature.

[TomKaltz]

Anyone care to implement this for the ffmpeg producer. The osc_monitor is already used inside that class so all you have to do is calculate the levels as and send to the monitor. Check core/mixer/audio/audio_mixer.cpp for clues. Again, we encourage someone in the community to just take a stab at it. Even a pull request with buggy code is better than nothing. Github allows us to all collaborate on fixing bugs before they are committed to the master branch.