PhoneVR icon indicating copy to clipboard operation
PhoneVR copied to clipboard

Hardware encoding and Latency minimization via FFMPEG

Open ShootingKing-AM opened this issue 4 years ago • 0 comments

Hardware Encoding Support and Network hand-wave latency reduction

  • [x] Research low-latency Protocols: Done: RTP best for this application.
  • [x] Research Available libraries
    • [x] PJ-SIP, FFMPEG
    • [x] PJ-SIP - Tests: Status: Failed no pj-media Support.
    • [ ] Implement FFMPEG
      • [ ] Implement FFMPEG on Desktop
        • [x] Protocols Selection: RTP (ultra-low latency) + NVENC/NVDEC Encoder (h.264 Container format)
        • [ ] Code Implementation:
          • [x] Implement ffmpeg as ffmpeg-cli commandLineInterface integration
            • [x] Stage0 - Research D3DSharedResourceHandle Streaming
              • [x] Test1: using Inprocess pipes to get pipe render to ffmpeg for encoding. Status: Failed - WritePipe() Speed too low (~15 fps)
              • [x] Test2: Piping D3DSharedResourceHandle to ffmpeg.exe process in background. Status Failed - Hard to maintain with FFMPEG-master updates, better to implement libav*
                • [x] ~Bypass Mobile negotiation, for Development~
                • [x] ~Modify FFMPEG fopen_inputs() to get D3DSharedResourceHandle~
          • [ ] Implement ffmpeg as libav* libraries, if Test1 & 2 Both fail.
            • [ ] Stage1 - Static Content Streaming
              • [ ] Stream Static testfile.mp4, with RTP to localhost, for Dev
                • [x] Bypass Mobile negotiation, for Development
                • [x] Code FFMPEG Init()
                • [x] Code FFMPEG fopen_inputs()
                • [x] Code FFMPEG fopen_outputs()
                • [x] Code FFMPEG transcode()
                  • [x] Code FFMPEG transcode_init()
                  • [x] Code FFMPEG transcode_step() <--- Main Loop
                • [ ] Code FFMPEG stop_stream(), Outputs and Inputs
                • [x] Code FFMPEG deInit() <--- safe exit + Memory Release
            • [ ] Stage2 - Dynamic Content Streaming
              • [ ] Modify FFMPEG fopen_inputs() to get D3DSharedResourceHandle
                • [x] Research methodology
                • [x] Modify FFMPEG fopen_inputs() to take D3DSharedResourceHandle as Input instead of Static file
      • [ ] Implement FFMPEG on Mobile End
        • [ ] Add shared libraries (.so) to Android project.
        • [ ] Research into available FFMPEGAndroid ports/wrappers
          • [ ] Test3: Port Wrapper Implementation test on Android
        • [ ] If Test3 not successful, implement FFMPEG libav* code.

Fixes #44 and #52

ShootingKing-AM avatar Dec 22 '20 10:12 ShootingKing-AM