I use fps filter, there are problems with sound

[vonabe]

The sound goes well for about 100 seconds and then breaks off.

    FFmpegFrameGrabber grabber = new FFmpegFrameGrabber("file.mkv");
    grabber.start();

    FFmpegFrameRecorder recorder = new FFmpegFrameRecorder("rtmp://live-fra.twitch.tv/app/live_", 1920, 1080, 2);
    recorder.setInterleaved(true);
    recorder.setVideoOption("tune", "zerolatency");
    recorder.setVideoOption("preset", "veryfast");
    recorder.setVideoOption("crf", "24");
    recorder.setVideoOption("threads", "2");
    recorder.setVideoBitrate(4_000_000);
    recorder.setPixelFormat(AV_PIX_FMT_YUV420P);
    recorder.setVideoCodec(avcodec.AV_CODEC_ID_H264);
    recorder.setAudioCodec(avcodec.AV_CODEC_ID_AAC);
    recorder.setFrameRate(30);
    recorder.setGopSize(60);
    recorder.setAudioQuality(0);
    recorder.setAudioBitrate(grabber.getAudioBitrate());
    recorder.setSampleRate(grabber.getSampleRate());
    recorder.setAudioChannels(grabber.getAudioChannels());
    recorder.setFormat("flv");
    recorder.start();

    String scale = recorder.getImageWidth() + ":" + recorder.getImageHeight();
    FFmpegFrameFilter filter = new FFmpegFrameFilter("fps=fps=30,scale=" + scale + ":force_original_aspect_ratio=1,pad=" + scale + ":(ow-iw)/2:(oh-ih)/2", "volume=2.5", grabber.getImageWidth(), grabber.getImageHeight(), 2);
    filter.setAudioChannels(grabber.getAudioChannels());
    filter.setSampleFormat(grabber.getSampleFormat());
    filter.setSampleRate(grabber.getSampleRate());
    filter.setFrameRate(grabber.getFrameRate());
    filter.start();

    grabber.setVideoFrameNumber(10000);
    grabber.setAudioFrameNumber(10000);

    FPSCounter fpsCounter = new FPSCounter(fps -> System.out.println("Frame Rate Push - " + fps));

    while (true) {
        Frame grab = grabber.grabFrame(true, true, true, false, false);
        if (grab != null) {
            filter.push(grab);
            Frame frame = filter.pull();
            if (frame != null) {
                recorder.record(frame);
            }
            fpsCounter.render();
        }
    }

[saudet]

You might want to try to call grabAtFrameRate() instead of grabFrame().

[vonabe]

You might want to try to call grabAtFrameRate() instead of grabFrame().

I checked it with a 30 fps filter - the sound cuts out after 100 seconds. With fps 60 immediately without sound. If you stream with original fps recorder.setSampleRate(grabber.getFrameRate()); there is no sound problem. I think the problem may be with the timestamp samples. And grabAtFrameRate allowed me to not push too often, as far as I understand.

while (true) {
        Frame grab = grabber.grabAtFrameRate();
        if (grab != null) {
            if (grab.image != null || grab.samples != null) {
                filter.push(grab);
            }
            Frame frame = filter.pull();
            if (frame != null) {
                recorder.record(frame);
                fpsCounter.render();
            }
        } else System.out.println("grub null stop");
    }

[saudet]

If you're not seeing any messages from FFmpeg on the console, make sure that FFmpegLogCallback.set() has been called. You might get more information about what is going wrong that way.

[vonabe]

If you're not seeing any messages from FFmpeg on the console, make sure that FFmpegLogCallback.set() has been called. You might get more information about what is going wrong that way.

I get warnings after the sound disappears Warning: [out @ 000001c243e53fc0] 100 buffers queued in out, something may be wrong. Warning: [out @ 000001c243e53fc0] 1000 buffers queued in out, something may be wrong.

[saudet]

Maybe you're not calling pull() often enough. Try to add a loop there.

[vonabe]

Maybe you're not calling pull() often enough. Try to add a loop there.

Yes it works thank you very much. I think this should be added to the examples. Is it posible to set h264 profile to FF_PROFILE_H264_MAIN, I saw in the source code the hardcode on video_c.profile(AVCodecContext.FF_PROFILE_H264_CONSTRAINED_BASELINE); ?

[saudet]

We can add more examples, we can also extend FFmpegFrameRecorder, yes that's all fine. Please open pull requests for that purpose. Contributions are welcome!

[vonabe]

@saudet Any idea how to set CBR?

[saudet]

Something else than setVideoBitrate()?

[vonabe]

Something else than setVideoBitrate()?

I set 5000000 bit rate in the statistics I see a reproducible bitrate of 1600 kbps and not very good quality.

[saudet]

You'll need to show me what you're doing to get this, because it's not doing that for me. I can't fix what isn't broken!

[vonabe]

@saudet Hello, how can i sync video frames with audio? I have images that I take as screenshots 30 times per second and the sound of mp3 from grabber, records it all in the FFmpegFrameRecorder, but when I stop the playback process, the video lasts another 5-10 seconds and the sound is stoped. I don't know how to sync.

[saudet]

Call setTimestamp() with the correct values before calling record().

[vonabe]

@saudet Can you tell me what these values are? I have frames without timestamp and audio with her timestamps. Сan you tell me how to generate timestamp for rtmp?

[saudet]

BTW, did you try the working example from issue https://github.com/bytedeco/javacv/issues/1398#issuecomment-613208633 ? If it works, please open a new pull request to add this code the the samples directory! Thanks

[saudet]

Yes it works thank you very much. I think this should be added to the examples. Is it posible to set h264 profile to FF_PROFILE_H264_MAIN, I saw in the source code the hardcode on video_c.profile(AVCodecContext.FF_PROFILE_H264_CONSTRAINED_BASELINE); ?

It's not actually hard coded. I've confirmed that, for example, calling setVideoOption("profile", "main") before start() does set the profile to "main", see https://trac.ffmpeg.org/wiki/Encode/H.264#Profile.

[saudet]

@saudet Any idea how to set CBR?

Similarly, we should be able to set that by calling setVideoOption("crf", "whatever").