ffmpegframerecorder.record() is taking long(sometimes 3-4s per frame) on android device

[streamcam7]

I have modified the RecordActivity on samples and i am able to do a full live stream (to a RTMP server). I see that record(frame) method, continually gets worse. I have tried timestamping the call time and initially it takes < 30 ms to begin with but gradually it hovers around excess of 400ms and stays at that for a while.

As i let the rtmp stream continue for longer, i end up getting time in excess of >3000 to 4000ms.

The problem i have is the frame drops i have to manage as Camera is spitting out avg of 30fps. FFmpegframerecorder.record() doesn't keep up with the production rate.

I am not sure if i am doing anything wrong ? Is it normal for record() to take that much time. I am not applying any filters to the frame. I am also managing the production and consumption rates with a fixed buffer, which is fine.

But how do i get around the excessive time record() is taking ? What am i missing ? I will try the preset option to ultrafast to see if it helps my case.

I am using Android R (11) image. Any Ideas or pointers regarding this will be helpful

[saudet]

Please first try to upgrade to JavaCV 1.5.6.

[streamcam7]

Thanks for the reply.

I am on JavaCV 1.5.6

Below is what i am using (i commented out platform as its increasing my apk size to close to 800+MBs)

//implementation group: 'org.bytedeco', name: 'javacv-platform', version: '1.5.6' implementation group: 'org.bytedeco', name: 'javacv', version: '1.5.6' implementation group: 'org.bytedeco', name: 'javacpp', version: '1.5.6', classifier: 'android-arm64' implementation group: 'org.bytedeco', name: 'ffmpeg', version: '4.4-1.5.6', classifier: 'android-arm64'

[saudet]

Ok, thanks, could you run a profiler on that to see which method call is taking all that time?

[streamcam7]

I am not sure what you mean by running profiler ? Do i have to enable certain logging ? FFmpegLogCallback.set() and some system props ?

Any pointers ? I'd be happy to look into this ..

[saudet]

That would be the "Method and function traces" here: https://developer.android.com/studio/profile/cpu-profiler Thanks!

[streamcam7]

Thank you. I will look into this and get back with my observations. Much appreciated

[streamcam7]

This is Summary of record () thread-

Also here is the exported trace file which i ran for about 15 secs (Approx) cpu-art-20210915T134518.zip

A few points which may be significant for context (in case you needed more details): (Overall whole streaming does work, but i am after performance issues (potentially) from record() and VLC lag (which i think may improve if use preset to ultrafast)

1. I am running 2 threads with executor (Audio and Video)
2. My Code is significantly different from sample record activity as i am working with camera2 api
3. I am receiving YUV_420_888 format from Camera, i am converting it to NV21 via native call before calling record(frame,NV21 format)
4. I have a producer consumer topology for receiving and consuming frames
5. i am not setting any timestamps on Audio frames/samples and simply calling
6. i am setting video frame timestamp to when they are received and stored in a blocking queue
7. If frames' time stamp is greater than that of recorder's time stamp, i set FrameRecorder's timestamp to frame's timestamp

here's the code to give clarity :

AudioThread: while (mThreadRunning) { if (mIsStreaming && mFrameRecorder != null) { bufferReadResult = mAudioRecord.read(mAudioData.array(), 0, mAudioData.capacity()); mAudioData.limit(bufferReadResult); if (bufferReadResult > 0) { Log.i(APP_TAG, CLASS_NAME+": bufferReadResult: " + bufferReadResult); try { mFrameRecorder.recordSamples(mAudioData); } catch (FFmpegFrameRecorder.Exception e) { Log.e(APP_TAG, e.getMessage()); e.printStackTrace(); } } } }

VideoThread:

if (mFrameRecorder != null) { long timestamp = frame.getTimeStamp(); //Allow proper counting synchronized (VideoRunnable.class) { // this is just a counter to track how many video frames are processed FramesProcessed++; //since only 1 video thread is running it wont matter } if (timestamp > mFrameRecorder.getTimestamp()) { mFrameRecorder.setTimestamp(timestamp); } try { long startTime = System.currentTimeMillis(); mFrameRecorder.record(frame.getFrame(),Integer.parseInt(mPixFMT)); long endTime = System.currentTimeMillis(); long processTime = endTime - startTime; mTotalFrameProcessTime = mTotalFrameProcessTime+ processTime; Log.i(APP_TAG, CLASS_NAME+": Time taken for Frame recorder record() " + processTime + "ms" +"<-- Thread ID:"+this.hashCode()); }catch (FFmpegFrameRecorder.Exception e) { Log.e(APP_TAG,CLASS_NAME+": Error Recording Video:"+e.getMessage() +": <-- Thread ID:"+this.hashCode()); e.printStackTrace(); } }

Any help/pointer is much appreciated..

[streamcam7]

I just noticed that Frame class has timestamp in microseconds, but i am assigning it the long value retrieved from System.currentTimeMillis(). I guess that would be problematic, wouldnt it ?

/** Timestamp of the frame creation in microseconds. */
public long timestamp;

[saudet]

Yes, that could cause problems...

[streamcam7]

I will fix it and rerun the traces

On Wed, Sep 15, 2021 at 2:43 PM Samuel Audet @.***> wrote:

Yes, that could cause problems...

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/bytedeco/javacv/issues/1693#issuecomment-920403514, or unsubscribe https://github.com/notifications/unsubscribe-auth/AVUF5J6GDS5WDSUMSBZ6I6LUCEHQDANCNFSM5EA6ZQ3A . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

[streamcam7]

Okay I fixed that timestamp issue and* added preset to ultrafast*. Now it shows significant improvement but still ends up taking in excess of 800ms in some instances. I have some optimization of my own to make, but just profiling record() method seems to have this issue.

To quantify a bit, out of roughly 400 frames I received , i see record() method was taking 600-800ms in about 20-30 instances on average. Which is way better that what was happening before.

Any pointers ?

Additionally, not that i want to mix up different issues but i am getting some warning system.err msgs (i am sure i can ignore these, just pointing in case they may have something in terms of optomizations i may be missing out on)

W/System.err(24715): Debug: Loading class org.bytedeco.javacpp.presets.javacpp Line 725: 09-20 13:04:28.498 W/System.err(24715): Debug: Loading class org.bytedeco.javacpp.Loader Line 734: 09-20 13:04:28.521 W/System.err(24715): Debug: Loading class org.bytedeco.javacpp.presets.javacpp Line 735: 09-20 13:04:28.522 W/System.err(24715): Debug: Loading class org.bytedeco.javacpp.Pointer Line 736: 09-20 13:04:28.525 W/System.err(24715): Debug: Loading class org.bytedeco.javacpp.presets.javacpp Line 737: 09-20 13:04:28.525 W/System.err(24715): Debug: Loading class org.bytedeco.ffmpeg.global.avutil Line 739: 09-20 13:04:28.533 W/System.err(24715): Debug: Loading class org.bytedeco.javacpp.presets.javacpp Line 740: 09-20 13:04:28.533 W/System.err(24715): Debug: Loading class org.bytedeco.ffmpeg.global.avutil Line 753: 09-20 13:04:28.581 W/System.err(24715): Debug: Loading class org.bytedeco.ffmpeg.avutil.LogCallback

On Wed, Sep 15, 2021 at 3:48 PM cam stream @.***> wrote:

I will fix it and rerun the traces

On Wed, Sep 15, 2021 at 2:43 PM Samuel Audet @.***> wrote:

Yes, that could cause problems...

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/bytedeco/javacv/issues/1693#issuecomment-920403514, or unsubscribe https://github.com/notifications/unsubscribe-auth/AVUF5J6GDS5WDSUMSBZ6I6LUCEHQDANCNFSM5EA6ZQ3A . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

[saudet]

To encode in real time on Android devices, you'll probably need to use the hardware accelerator, but unfortunately, FFmpeg doesn't support that on Android, yet. You'll need to use directly the MediaCodec API from Android for that, see https://github.com/bytedeco/javacv/issues/945#issuecomment-405900044.

/cc @tmm1

[streamcam7]

Ok, thanks. That's a bummer. I'll see what i can do.

On Mon, Sep 20, 2021 at 5:19 PM Samuel Audet @.***> wrote:

To encode in real time on Android devices, you'll probably need to use the hardware accelerator, but unfortunately, FFmpeg doesn't support that on Android, yet. You'll need to use directly the MediaCodec API https://developer.android.com/reference/android/media/MediaCodec from Android for that, see #945 (comment) https://github.com/bytedeco/javacv/issues/945#issuecomment-405900044.

/cc @tmm1 https://github.com/tmm1

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/bytedeco/javacv/issues/1693#issuecomment-923463295, or unsubscribe https://github.com/notifications/unsubscribe-auth/AVUF5J7RE6NPNM3GU3EAU63UC7FQLANCNFSM5EA6ZQ3A . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

[streamcam7]

Just out of curiosity, i am not native level developer,as i work mainly on App side. What work would it entail to make this work with MediaCodec HWACCL (not involving ffmpeg). And also the option for ffmpeg HwAccel support.

On Mon, Sep 20, 2021 at 5:52 PM cam stream @.***> wrote:

Ok, thanks. That's a bummer. I'll see what i can do.

On Mon, Sep 20, 2021 at 5:19 PM Samuel Audet @.***> wrote:

To encode in real time on Android devices, you'll probably need to use the hardware accelerator, but unfortunately, FFmpeg doesn't support that on Android, yet. You'll need to use directly the MediaCodec API https://developer.android.com/reference/android/media/MediaCodec from Android for that, see #945 (comment) https://github.com/bytedeco/javacv/issues/945#issuecomment-405900044.

/cc @tmm1 https://github.com/tmm1

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/bytedeco/javacv/issues/1693#issuecomment-923463295, or unsubscribe https://github.com/notifications/unsubscribe-auth/AVUF5J7RE6NPNM3GU3EAU63UC7FQLANCNFSM5EA6ZQ3A . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

[saudet]

Like I said above https://github.com/bytedeco/javacv/issues/1693#issuecomment-923463295, it's still possible to use MediaCodec from the application level in Java using the Android API.

[tmm1]

There is some work being done to add MediaCodec based hardware-encoding to FFmpeg: https://patchwork.ffmpeg.org/project/ffmpeg/patch/[email protected]/

[tmm1]

FFmpeg 6.0 has been released and includes support for hardware encoding on Android

[saudet]

FFmpeg 6.0 has been released and includes support for hardware encoding on Android

Awesome! I've updated the presets and JavaCV.

@streamcam7 Please give it a try with the snapshots: http://bytedeco.org/builds/

[sprigogin]

What is ETA for the stable release of 6.0-1.5.9?

[saudet]

Version 1.5.9 has been released! Enjoy