go-livepeer icon indicating copy to clipboard operation
go-livepeer copied to clipboard

[WIP] AI video prototype

Open yondonfu opened this issue 1 year ago • 2 comments

Opening this draft PR to kick off CI build process for the WIP ai-video branch. This PR should not be merged as-is and the code on this branch will likely be refactored + cleaned up separately later on.

yondonfu avatar Jan 23 '24 22:01 yondonfu

https://github.com/livepeer/go-livepeer/pull/2959/commits/d32b1b57d569f5e542e73f2395a97498e73fbfef temporarily disables Linux arm64 builds because they fail due to an error related to not being able to find zlib during ffmpeg compilation. This error doesn't occur for amd64 builds so I suspect the issue is related to the amd64 -> arm64 cross-compilation process. I noticed that we compile an arm64 specific version of x264 before compiling ffmpeg - perhaps we have to do something similar with zlib?

zlib is currently required as a dependency as of https://github.com/livepeer/go-livepeer/pull/2959/commits/133050de538765613d499f4f8e0b911fed8e0ae6#diff-4ae778054809274731b9da0c6a5a869c0bd214e92f954a5c9c39181748c2f175 which enabled the png decoder and image2 muxer which are used to demux + decode a sequence of PNG files so they can be encoded into an mp4 file. Ideally, we would replace the PNG demux/decode component by passing tensors (that represent frames) outputted by a model directly from GPU memory to NVENC using torchaudio.StreamWriter, but torchaudio.StreamWriter doesn't support RGB -> YUV conversion on the GPU yet - it can still encode a larger, less-streaming friendly (my understanding is yuv420p is preferred for streaming) RGB output, but I didn't jump to implement this yet due to current limitations. Until this replacement happens, zlib would be a required dependency to support the temporary PNG demux + decode component.

yondonfu avatar Jan 24 '24 17:01 yondonfu

@yondonfu Weird - on release go-livepeer right now zlib is dynamically linked. I'll have a look.

iameli avatar Jan 24 '24 18:01 iameli