Adding audios renders black playerItem if tracks have transitions

[kicortez]

I'm creating a video editing app and I have functionality to add transitions and audios.

When adding transitions, without added audios, it renders fine, with transitions and all.

Also when adding audios, without added transitions, it renders fine, audio plays fine.

However, if I add a transition and audio on the timeline, only the audio plays and the video is only black. Here's my code.

private func buildTracks() {
	var videoChannel: [TrackItem] = []
	var audioChannel: [TrackItem] = []
	
	for asset in assets {
		let resource = trackResource(for: asset)

		let trackItem = TrackItem(resource: resource)
		trackItem.videoConfiguration.contentMode = .aspectFit
		
		switch asset.transition {
		case 1:
			let transitionDuration = CMTime(seconds: 0.5, preferredTimescale: preferredTimeScale)
			let transition = CrossDissolveTransition(duration: transitionDuration)
			trackItem.videoTransition = transition
			print("CROSS DISSOLVE")
		case 2:
			let transitionDuration = CMTime(seconds: 0.5, preferredTimescale: preferredTimeScale)
			let transition = FadeTransition(duration: transitionDuration)
			trackItem.videoTransition = transition
			print("FADE BLACK")
		case 3:
			let transitionDuration = CMTime(seconds: 0.5, preferredTimescale: preferredTimeScale)
			let transition = FadeTransition(duration: transitionDuration)
			trackItem.videoTransition = transition
			print("FADE WHITE")
		default:
			trackItem.videoTransition = nil
			print("NONE")
		}
		
		if let asset = asset as? VVideoAsset {
			trackItem.audioConfiguration.volume = asset.volume
		}
		
		videoChannel.append(trackItem)
		audioChannel.append(trackItem)
		
		let filterConfigurations = videoEdit.filters.map { FilterConfiguration(filter: $0, totalDuration: totalDuration) }
		trackItem.videoConfiguration.configurations = filterConfigurations
	}
	
	timeline.videoChannel = videoChannel
	timeline.audioChannel = audioChannel
}

private func buildAudios() -> [AudioProvider] {
	var audios: [AudioProvider] = []
	
	videoEdit.audios.forEach { (audio) in
		guard let audioURL = audio.audioAsset.mp3Path else {
			return
		}
		
		let audioAsset = AVAsset(url: audioURL)
		let resource = AVAssetTrackResource(asset: audioAsset)
		let duration = audio.duration * totalDuration
		
		resource.selectedTimeRange = CMTimeRange.init(start: CMTime.zero, end: CMTimeMakeWithSeconds(duration, preferredTimescale: preferredTimeScale))
		let audioTrackItem = TrackItem(resource: resource)
		audioTrackItem.audioConfiguration.volume = audio.volume
		audioTrackItem.startTime = CMTimeMakeWithSeconds(audio.componentStart * totalDuration, preferredTimescale: preferredTimeScale)
		
		audios.append(audioTrackItem)
	}
	
	return audios
}

private func buildAddedComponents() {
	timeline.audios = buildAudios()
	timeline.overlays = buildOverlays()
}

[kartuzovmax]

Same problem

[DK-L-iOS]

Same problem

[Coder-ACJHP]

Did you solve this problem, because still I am getting same result like yours. I think this is timing issue, when we add transition to track item the audio time is being longer then the video because of transition is shorting trackItem time.

[Coder-ACJHP]

Actually transition should not shorten video duration because it's happening between 2 items. Transition time must be created between these item [prev item] T [next item] ||||||||||||-----------||||||||||| I mean transition timing should be something like this:

let prevItem: TrackItem = ....
let nextItem: TrackItem = ....
        
let transitionDuration = CMTime(seconds: 5, preferredTimeScale: 600)
let halfOfTransitionDuration = CMTimeMultiplyByRatio(transitionDuration, multiplier: 1, divisor: 2)

// Half of transition duration cut it from prevItem
let prevItemNewDuration = prevItem.timeRange.duration.seconds - halfOfTransitionDuration
prevItem.timeRange.duration = CMTime(seconds: prevItemNewDuration, preferredTimeScale: 600)

// And other half from nextItem
let nextItemNewStartTime = nextItem.timeRange.startTime.seconds + halfOfTransitionDuration
nextItem.timeRange.startTime = CMTime(seconds: nextItemNewStartTime, preferredTimeScale: 600)
let nextItemNewDuration = nextItem.timeRange.duration.seconds - halfOfTransitionDuration
nextItem.timeRange.duration = CMTime(seconds: nextItemNewDuration, preferredTimeScale: 600)

In this case video duration will not change and audio timing issue will not happen with transitions. I hope @vitoziv find a solution for transition timing issue to avoid black screen problem with custom audios

[Coder-ACJHP]

For temporary solution we can subtract total transitions duration from video duration, I am using the below code with custom audio + overlay video + transitions + custom effects all together works well with this way:

let resource = AVAssetTrackResource(asset: AVAsset(url: mediaURL))
// My transitions duration is 1.0 so I can subtract transitions list element count from video duration.
// 'timelineComposer' is timeline builder
let audioDuration = timelineComposer.timeline.totalVideoDuration - CGFloat(timelineComposer.getTransitionList().count)
resource.selectedTimeRange = CMTimeRange(start: .zero, duration: CMTime(seconds: audioDuration, preferredTimescale: 600))
let trackItem = TrackItem(resource: resource)
trackItem.audioConfiguration.volume = 1.0  
timelineComposer.addAudio(item: trackItem)
// Build your timeline ....

[vitoziv]

Hey guys, if we add a video transition, we also need to add an audio transition with the same duration, it's an implicit requirement, I know it's not a good API design. you can temporarily add an audio transition to fix this problem.

see the demo code: https://github.com/VideoFlint/Cabbage/blob/41c78700c32b3814eaa242a3b4fe9323e14d63fc/Cabbage/ViewController.swift#L248-L250

Recently I will have some free time to continue maintaining this project, this project has actually moved a big step in these 2 years, there are many API optimize and many compatibility issues fixes, I will release Cabbage 2.0 within 4 weeks, most of issues will be fixed in Cabbage 2.0