How can i combine video and audio

[lianganwei1994]

Thanks for provide this convenient project which help me a lot,but I've encounter a big trouble as folw: process1 = ( ffmpeg .input("XXX") .output('pipe:', format='rawvideo', pix_fmt='rgb24',video_bitrate=1244160, vframes=100) .run_async(pipe_stdout=True) ) here in your example code I notice that you input the video to pipe and then convert it to array so that you can handle it by the while loop: while True: XXXX XXXX the loop read the bytes from process1 and send them to pipe,So it's going to be video. Although it's a good idea,but I found the audio has lost. process2 = ( ffmpeg .input('pipe:', format='rawvideo', pix_fmt='rgb24', s='{}x{}'.format(width, height)) .output('/home/material/lianganwei/test/transform.mp4', video_bitrate=1244160,pix_fmt='rgb24') .overwrite_output() .run_async(pipe_stdin=True) ) My trouble is that how to combine the audio and video in process2? I've tried so many method but dosen't work.

[simfeng]

same problem with u

[WillLiGitHub]

plus 1

[MarSoft]

I have the same problem too. But the actual problem is even worse; you not only need to split audio and video streams, but you also need to keep them in sync with each other. So you should somehow get video frame timestamps as well as audio timestamps. Raw video frames don't include any timestamps. Probably you are best using libavcodec and other libraries behind ffmpeg. I think there should be some Python wrappers for them as well.

[AnoyiX]

+1

[LeonLIU08]

+1

[simlmx]

I managed to do it with

import ffmpeg
import numpy as np

INPUT = "in.mp4"
OUTPUT = "out.mp4"


# Probe the input video.
probe = ffmpeg.probe(INPUT)
info = next(s for s in probe["streams"] if s["codec_type"] == "video")
width = info["width"]
height = info["height"]
avg_frame_rate = info["avg_frame_rate"]


# Define the input and output processes.
# The output process is a video and audio stream concatenated together.
input_process = (
    ffmpeg.input(INPUT)
    .output("pipe:", format="rawvideo", pix_fmt="rgb24")
    .run_async(pipe_stdout=True)
)

video_output = ffmpeg.input(
    "pipe:",
    format="rawvideo",
    pix_fmt="rgb24",
    s="{}x{}".format(width, height),
    r=avg_frame_rate,
)

audio_output = ffmpeg.input(INPUT).audio

output_process = (
    ffmpeg.concat(video_output, audio_output, v=1, a=1)
    .output(OUTPUT, pix_fmt="yuv420p")
    .overwrite_output()
    .run_async(pipe_stdin=True)
)


# Run the processes.
while True:
    in_bytes = input_process.stdout.read(width * height * 3)
    if not in_bytes:
         break
     frame = np.frombuffer(in_bytes, np.uint8).reshape([height, width, 3])

     frame = frame.copy()
     frame[:, :, 0] = 0

     output_process.stdin.write(frame.astype(np.uint8).tobytes())

 output_process.stdin.close()
 input_process.wait()
 output_process.wait()

[nile649]

	def _saveVideo_ffmpeg_(videourl:str,audiourl:str,title:str,path:str):   
		try:
			video = ffmpeg.input(videourl)
			audio = ffmpeg.input(audiourl)
			ffmpeg.concat(video, audio, v=1, a=1).output(''.join([path,title,".mp4"])).run()
		except ValueError as e:
			logger.exception(e)

[ddesimio]

anyone have success with this when receiving one frame (numpy array) of audio and video data at a time?