[MovieInput/MovieOutput/SpeakerOutput] Audio support, synchronized encoding, internal movie recording dispatch queue context support, looping, pausing, encoding progress, real-time threads (optional), AVVideoComposition support, resolve silently dropped frames, etc.

[joshbernfeld]

NOTE: If you are looking to use this I wrote Pixel SDK which includes GPUImage and this pull request via CocoaPods.

Otherwise, this pull request should be used with pr https://github.com/BradLarson/GPUImage2/pull/243 and they can be found together in my fork in addition to the above SDK.

MovieInput

• Added synchronized video encoding support (filter an existing movie file and save the result to a new movie file). Synchronized reading works by blocking the reading thread with an NSCondition lock when both inputs are unable to accept new data. This allows for processing logic of both synchronized and unsynchronized reading to be shared vs using requestMediaDataWhenReadyOnQueue separately. If you need to debug synchronized encoding for a problematic piece of media a synchronizedEncodingDebug var has been added.

• Added SimpleVideoEncoder Xcode project and README example for synchronized video encoding. Note: All audio settings have been left up to the user. The Xcode project example shows how to use passthrough and AAC encoding but the README just shows AAC audio encoding for simplicity.

• Updated SimpleVideoRecorder Xcode project to properly handle audio.

• Added sound playback support via new SpeakerOutput class by processing audio samples retrieved from AVAssetReader.

• Added looping playback.

• Added pausing and start(atTime:)

• Added synchronized video encoding progress block.

• Added AVVideoComposition playback.

• Updated playAtActualSpeed algorithm to use a timeline based algorithm instead of time spent algorithm which uses nanoseconds and mach_wait_until(). This improves playback speed/accuracy for 60fps video and is necessary for correctly aligned audio playback.

• Added real-time threads support w/ mach_wait_until() for smoother playback of 60fps video (optional). In order to get their full benefit, you probably need to do all of the work on the one thread instead of syncing with the sharedImageProcessingContext but that just isn't realistic here. Real-time threads are turned off by default and a useRealtimeThreads var has been added. In addition the sharedImageProcessingContext queue priority has been made one level higher for improved playback quality.

• Added support for custom audioSettings, no defaults are applied and it is up to the user to provide their own values. A nil value for audioSettings is equivalent to pass-through audio.

• Added support for frame reprocessing. If the video is paused you can call transmitPreviousFrame() in order to forward the last framebuffer through the pipeline again.

• Fixed AVAssetReader concurrent access crashes. A call to cancelReading() was occasionally causing exc_bad_access since it was not done on the thread responsible for the asset reader. Now each asset reader has its own thread and only that thread has access to the asset reader. Using a single thread for each asset reader improves stability, allows for the use of real-time threads, and simplifies synchronized reading logic. In addition, the assetReader instance variable has been removed to prevent accidentally accessing it from the main thread.

• Removed asset.loadValuesAsynchronously() since the dedicated thread will just get blocked when the track is being loaded in createAssetReader().

• Added support for catching objective c exceptions thrown on startReading() and print them. This resolves hard crashes caused by problematic video files. In addition, print the asset reader error for when startReading() returns false.

• Note markAsFinished() is now only used during synchronized encoding because it is usage appears only intended for when we are monitoring isReadyForMoreMediaData to control the flow of data. I observed that when markAsFinished was in place on live videos sometimes they will end 2 or 3 seconds earlier than normal.

  If you are monitoring each input's isReadyForMoreMediaData value to keep the output file well interleaved, it is important to call this method when you have finished adding buffers to a track. This is necessary to prevent other inputs from stalling, as they may otherwise wait forever for that input's media data, attempting to complete the ideal interleaving pattern.

  https://developer.apple.com/documentation/avfoundation/avassetwriterinput/1390122-markasfinished

  Here is an answer which supports that too https://stackoverflow.com/a/4437047/1275014

• Added shouldInvalidateAudioSampleWhenDone to the AudioEncodingTarget protocol.

• Fixed black frames at the end of videos by ending the session with the last received frame. If we only call finishWriting() the session's effective end time will be the latest end timestamp of the session's samples which could be either video or audio.

  You do not need to call [endSession(atSourceTime:)]; if you call finishWriting without calling this method, the session's effective end time will be the latest end timestamp of the session's samples (that is, no samples will be edited out at the end).

  https://developer.apple.com/documentation/avfoundation/avassetwriter/1389921-endsessionatsourcetime

MovieOutput

• Added synchronized video encoding support.

• Added internal movie recording dispatch queue context support.

• Fixed dropped CVPixelBuffers

  Do not modify a CVPixelBuffer or its contents after you have passed it to this method.

  https://developer.apple.com/documentation/avfoundation/avassetwriterinputpixelbufferadaptor/1388102-append

  It must then be a requirement that for each frame we create a new pixel buffer with CVPixelBufferPoolCreatePixelBuffer(). Since we are not doing this, occasionally old pixel buffers are being reused. This is causing dropped frames - in the form of duplicates. These would mostly occur towards the beginning of videos and some would be interspersed throughout the rest of the video. I also noticed this problem impacted less powerful devices more. With this synchronized video encoding is now a 1-1 mapping of frames and 30/60fps video is written with no issue. This also fixes dropped frames when writing to file from the Camera.

  Related issue: https://github.com/BradLarson/GPUImage/issues/1501

• Added support for providing a naturalTimeScale to the asset writer. You should provide a naturalTimeScale if you have one for the current media. Otherwise the asset writer will choose one for you and it may result in misaligned frames if its not correct.

• Added support for custom audioSettings, no defaults are applied and it is up to the user to provide their own values. A nil value for audioSettings is equivalent to pass-through audio.

• Added support for catching objective c exceptions thrown on startWriting() and append() and print them. In addition, print the asset reader error for when startWriting() returns false.

SpeakerOutput

• Added sound playback support for MovieInput by processing audio samples retrieved from AVAssetReader. Related issue: https://github.com/BradLarson/GPUImage2/issues/205 Related issue: https://github.com/BradLarson/GPUImage2/issues/179

• Updated README and Xcode project SimpleMovieFilter example for audio playback, make sure your phone is not on silent when testing.

• Added readyForNextAudioBuffer() to the AudioEncodingTarget protocol so the SpeakerOutput can control the flow of sample buffers.

Camera

• Note: I included the Camera changes in this pull request since it has the updated AudioEncodingTarget protocol conformance.

• Removed call to removeAudioInputsAndOutputs() since it causes a significant black flash on video output when starting and stopping recording (aka from changing the audioEncodingTarget to a new MovieOutput). Adding the inputs and outputs just once seems to work fine for when the audioEncodingTarget gets changed in the future. This bug is reproducible in the SimpleVideoRecorder example. Brad if you have a reason for resetting the inputs and outputs each time the audioEncodingTarget is changed let me know. Related issue: https://github.com/BradLarson/GPUImage2/issues/137

• Added public access level to AVCapture related vars for users who wish to control flash and other additional functionality.

• Updated delegate var to weak to prevent retain cycle.

• Fixed deinit() crash if videoOutput failed to initialize in init().

• Fixed issue where capture sessions would sometimes randomly stop or fail to start by allowing for 1 restart attempt.

@Bradlarson If you have interest in merging this and want me to make changes - even syntax, write the OS X version, do certain tests, close open issues, or experience any problems please let me know. The same applies for the other pull requests I have submitted. I am also available to talk on the phone. The test devices I used where iPhone 5, iPhone 6, iPhone 6s, and iPhone X on iOS 10 and 11. I also tested using the Leaks profile in instruments. This is a pretty substantial pull request so I have no expectation you will merge this but in that vein I do believe this is a more stable and simple approach.

Note: I can only stand by the stability of this pull request if it is used in combination with pr https://github.com/BradLarson/GPUImage2/pull/243 and they can be found together here.

Real-time threads sources

Original thread configuration code:

Example applications that are candidates for high precision timers are…daemons that are streaming data to hardware with limited buffering, such as audio or video data.

https://developer.apple.com/library/content/technotes/tn2169/_index.html

Swift version of thread configuration code: https://stackoverflow.com/questions/45120492/how-do-i-achieve-very-accurate-timing-in-swift

Real-time thread details: https://developer.apple.com/library/content/documentation/Darwin/Conceptual/KernelProgramming/scheduler/scheduler.html#//apple_ref/doc/uid/TP30000905-CH211-BABCHEEB

Real-time thread parameter details: http://docs.huihoo.com/darwin/kernel-programming-guide/scheduler/chapter_8_section_4.html https://opensource.apple.com/source/xnu/xnu-344/osfmk/mach/thread_policy.h

Timing comparison info: https://mishravinay.wordpress.com/2013/09/19/idev-experiments-with-precise-timing-in-ios/

Queues vs threads:

Even though queues offer ways to configure the execution priority of tasks in the queue, higher execution priorities do not guarantee the execution of tasks at specific times. Therefore, threads are still a more appropriate choice in cases where you need minimal latency, such as in audio and video playback.

https://developer.apple.com/library/content/documentation/General/Conceptual/ConcurrencyProgrammingGuide/ThreadMigration/ThreadMigration.html

https://www.pixelsdk.com

[waterskier2007]

@BradLarson any thoughts on this PR? The corrupted image issue is still impacting the library for me.

[nhathm]

@joshbernfeld I used your sample SampleMovieEncoding, record a video with iPhone 7+ (iOS 12.x), then import to project then apply filter and encoding video.

But I faced this bug:

WARNING: Trouble appending pixel buffer at time: CMTime(value: 3092, timescale: 600, flags: __C.CMTimeFlags(rawValue: 1), epoch: 0) Optional(Error Domain=AVFoundationErrorDomain Code=-11800 "The operation could not be completed" UserInfo={NSLocalizedFailureReason=An unknown error occurred (-16341), NSLocalizedDescription=The operation could not be completed, NSUnderlyingError=0x2818ca670 {Error Domain=NSOSStatusErrorDomain Code=-16341 "(null)"}})

Video after filtered and exported's duration only have about 6 seconds. File: MovieOutput.swift Method: newFramebufferAvailable(_ framebuffer:Framebuffer, fromSourceIndex:UInt) Line of code: self.assetWriterPixelBufferInput.append(self.pixelBuffer!, withPresentationTime:frameTime)

Do you have any suggestion to fix this issue?

Thanks,

More info: if I create MovieOutput instance with this config, it works:

movieOutput = try MovieOutput(URL: exportedURL, size:Size(width:Float(videoTrack.naturalSize.width), height:Float(videoTrack.naturalSize.height)), fileType:.mov, liveVideo:false, videoSettings:videoEncodingSettings, videoNaturalTimeScale:videoTrack.naturalTimeScale, audioSettings:audioEncodingSettings, audioSourceFormatHint:audioSourceFormatHint)

Use fileType: mov insteand of mp4.

[joshbernfeld]

@nhathm according to this the solution might be to set a larger movieFragmentInterval. I would try passing nil to the videoNaturalTimeScale: parameter to see if that fixes it. Can you attach the video file you are trying to re-encode?

[nhathm]

@joshbernfeld I set nil to videoNaturalTimeScale, now movieFragmentInterval = CMTimeMakeWithSeconds(1, 1000) but still error.

Maybe because this file is HEVC codec? File: https://drive.google.com/file/d/1FWsIDPJ5Im4CeZ5vZ3X4XUyhX5EQUESb/view?usp=sharing

File info: Codecs: HEVC, AAC, Timed Metadata Audio: mono

Change file type to .mp4: not good Using .mov: OK

Thanks for your answer.

[joshbernfeld]

Exporting to HEVC also fixes the issue. Change the videoEncodingSettings to the following:

let videoEncodingSettings:[String:Any] = [
    AVVideoCompressionPropertiesKey: [
        AVVideoExpectedSourceFrameRateKey:videoTrack.nominalFrameRate,
        AVVideoAverageBitRateKey:videoTrack.estimatedDataRate,
        AVVideoProfileLevelKey:kVTProfileLevel_HEVC_Main_AutoLevel,
        AVVideoAllowFrameReorderingKey:videoTrack.requiresFrameReordering],
    AVVideoCodecKey:AVVideoCodecType.hevc.rawValue]

You will also need to change the iOS Deployment Target of the project to iOS 11 and add the following import statement:

import VideoToolbox

[nhathm]

@joshbernfeld thanks for your answer, it worked.

I really appreciate your help.