opencv_contrib
opencv_contrib copied to clipboard
opencv
cv2.cudacodec.createVideoReader error when input is a video stream
System information (version)
- OpenCV => 4.5.2
- Operating System / Platform => linux
- Compiler => gcc
Detailed description
I built opencv_contrib_python by myself. Here is the build information
Extra dependencies: m pthread cudart_static dl rt nppc nppial nppicc nppidei nppif nppig nppim nppist nppisu nppitc npps cublas cudnn cufft -L/usr/local/cuda/lib64 -L/lib64
3rdparty dependencies:
OpenCV modules:
To be built: calib3d core cudaarithm cudabgsegm cudacodec cudafeatures2d cudafilters cudaimgproc cudalegacy cudaobjdetect cudastereo cudawarping cudev dnn features2d flann highgui imgcodecs imgproc ml objdetect photo python3 stitching video videoio ximgproc
Disabled: aruco bgsegm bioinspired ccalib datasets dnn_objdetect dnn_superres dpm face fuzzy hfs img_hash intensity_transform line_descriptor mcc optflow phase_unwrapping plot quality rapid reg rgbd saliency shape stereo structured_light superres surface_matching text tracking videostab wechat_qrcode world xfeatures2d xobjdetect xphoto
Disabled by dependency: cudaoptflow
Unavailable: alphamat cnn_3dobj cvv freetype gapi hdf java julia matlab ovis python2 sfm ts viz
Applications: -
Documentation: NO
Non-free algorithms: NO
GUI:
GTK+: NO
Media I/O:
ZLib: /usr/local/lib/libz.so (ver 1.2.12)
JPEG: /usr/local/lib/libjpeg.so (ver 90)
WEBP: build (ver encoder: 0x020f)
PNG: /usr/local/lib/libpng.so (ver 1.6.37)
TIFF: build (ver 42 - 4.2.0)
JPEG 2000: build (ver 2.4.0)
HDR: YES
SUNRASTER: YES
PXM: YES
PFM: YES
Video I/O:
DC1394: NO
FFMPEG: YES
avcodec: YES (58.91.100)
avformat: YES (58.45.100)
avutil: YES (56.51.100)
swscale: YES (5.7.100)
avresample: NO
GStreamer: NO
v4l/v4l2: YES (linux/videodev2.h)
Parallel framework: pthreads
Trace: YES (with Intel ITT)
Other third-party libraries:
VA: NO
Lapack: NO
Eigen: NO
Custom HAL: NO
Protobuf: build (3.5.1)
NVIDIA CUDA: YES (ver 11.0, CUFFT CUBLAS NVCUVID)
NVIDIA GPU arch: 35 37 50 52 60 61 70 75 80
NVIDIA PTX archs:
cuDNN: YES (ver 8.1.1)
I want to use cuda to accelerate video decoding, but only found it worked on files. I want to use it on video stream.
Steps to reproduce
import cv2
url="rtsp://admin:[email protected]/media/video1"
cpu:
cv2.VideoCapture(url).read() # worked
gpu:
cv2.cudacodec.createVideoReader(url).next_frame()
get
opencv_contrib/modules/cudacodec/src/video_reader.cpp:128: error: (-210:Unsupported format or combination of formats) Unsupported video source in function 'nextFrame'
but if the url is a video file both cpu and gpu worked.
Issue submission checklist
- [x] I report the issue, it's not a question
- [x] I checked the problem with documentation, FAQ, open issues, forum.opencv.org, Stack Overflow, etc and have not found any solution
- [x] I updated to the latest OpenCV version and the issue is still there
- [x] There is reproducer code and related data files: videos, images, onnx, etc
Also i try 4.5.5 ,failed again @cudawarped here is the debug msg
[OPENCV:FFMPEG:40] Starting connection attempt to xxx port 8554
[OPENCV:FFMPEG:40] Successfully connected to xxx port 8554
[OPENCV:FFMPEG:40] SDP:
v=0
o=- 0 0 IN IP4 127.0.0.1
s=Stream
c=IN IP4 0.0.0.0
t=0 0
m=video 0 RTP/AVP 96
a=rtpmap:96 H264/90000
a=fmtp:96 packetization-mode=1; sprop-parameter-sets=Z2QAH6zZQFAFuwEQAAADABAAAAMDIPGDGWA=,aOvjyyLA; profile-level-id=64001F
a=control:trackID=0
[OPENCV:FFMPEG:40] setting jitter buffer size to 0
[OPENCV:FFMPEG:40] Reinit context to 1280x720, pix_fmt: yuv420p
Also get
cv2.error: OpenCV(4.5.5) modules/cudacodec/src/video_reader.cpp:140: error: (-210:Unsupported format or combination of formats) Unsupported video source in function 'internalGrab'
@wqh17101
I don't have any issues streaming from h264[5] RTSP sources. My guess would be the codec isn't supported or it has something to do with the way the parameter set is included.
Is it possible to give me a sample of the video using
ffmpeg -i "rtsp://admin:[email protected]/media/video1" -c copy sample.264
i push the stream by
ffmpeg -re -stream_loop -1 -i local_test.mp4 -vcodec libx264 -f flv -f rtsp rtsp://admin:[email protected]/stream/
and pull by
ffmpeg -hwaccel cuda -c:v h264_cuvid -hwaccel_output_format cuda -i rtsp://admin:[email protected]/stream/ -c:v h264_nvenc -b:v 5M output.mp4 -y
@cudawarped
Which verson of ffmpeg are you using, I cannot send and recieve a stream using those exact commands?
Have you tried streaming from other rtsp sources, live555, vlc etc.?
I am not sure why I cannot push an rtsp stream using your command
ffmpeg -re -stream_loop -1 -i local_test.mp4 -vcodec libx264 -f flv -f rtsp rtsp://admin:[email protected]/stream/
so lets try this another way. Can you use VideoCapture to get the SPS PPS info and the header of the first raw frame so I can see what is being passed to the decoder.
import cv2 as cv
import numpy as np
np.set_printoptions(formatter={'int':hex})
url="rtsp://admin:[email protected]/media/video1"
cap = cv.VideoCapture(url)
cap.set(cv.CAP_PROP_FORMAT,-1)
if(cap.get(cv.CAP_PROP_FORMAT) !=-1): print('Unable to activate raw bitstream reads')
iSpsPps = int(cap.get(cv.CAP_PROP_CODEC_EXTRADATA_INDEX))
ret, spsPps = cap.retrieve(flag=iSpsPps)
if(not ret): print('Unable to retrieve parapeter sets')
ret, encodedFrame = cap.read()
if(not ret): print('Unable to retrieve encoded frame')
print(f'SPS and PPS\n {spsPps[0]}')
print(f'Start of encoded frame\n {encodedFrame[0,:100]}')
The above should output something similar to
SPS and PPS [0x0 0x0 0x0 0x1 0x67 0x64 0x0 0x32 0xac 0xd2 0x0 0xa2 0x3 0xd7 0xe5 0x84 0x0 0x0 0xf 0xa4 0x0 0x3 0xe 0x8 0x10 0x0 0x0 0x0 0x1 0x68 0xea 0x8f 0x2c] Start of encoded frame [0x0 0x0 0x0 0x1 0x67 0x64 0x0 0x32 0xac 0xd2 0x0 0xa2 0x3 0xd7 0xe5 0x84 0x0 0x0 0xf 0xa4 0x0 0x3 0xe 0x8 0x10 0x0 0x0 0x0 0x1 0x68 0xea 0x8f 0x2c 0x0 0x0 0x0 0x1 0x65 0xb8 0x40 0x2f 0x6a 0xc6 0xa6 0x3 0xff 0x84 0x9a 0x51 0x6d 0x55 0x55 0x67 0xe9 0xd 0x52 0x14 0xed 0x33 0x3c 0xa7 0x88 0x70 0x2c 0xd0 0x6 0x29 0x8e 0xe0 0x0 0xf 0xec 0xe4 0x21 0x5e 0x1e 0xc6 0xd5 0x63 0x76 0x48 0xb7 0xb9 0x7e 0x0 0x18 0x51 0xe7 0x2 0x3f 0x1b 0x7c 0x8 0x21 0xa8 0x50 0xa0 0xe9 0x61 0x1b]
Here
[h264 @ 0x55c913ea2440] Missing reference picture, default is 0
[h264 @ 0x55c913ea2440] co located POCs unavailable
SPS and PPS
[0x0 0x0 0x0 0x1 0x67 0x64 0x0 0x1f 0xac 0xd9 0x40 0x50 0x5 0xbb 0x1 0x10
0x0 0x0 0x3 0x0 0x10 0x0 0x0 0x3 0x3 0x20 0xf1 0x83 0x19 0x60 0x0 0x0 0x0
0x1 0x68 0xeb 0xe3 0xcb 0x22 0xc0]
Start of encoded frame
[0x0 0x0 0x0 0x1 0x1 0x9e 0x3a 0x6a 0x46 0x7f 0x3b 0xaa 0x41 0xb4 0x3c
0x36 0xc1 0xf3 0xf2 0xf2 0x51 0x81 0x7 0x74 0xa0 0x5a 0xfe 0x80 0x31 0xa
0x41 0xc 0x8 0x55 0x61 0xac 0xc 0xf0 0x7a 0xe9 0x84 0xca 0x8c 0x73 0xc2
0xa9 0xb3 0xc1 0x1 0x29 0xca 0x4 0x21 0xec 0xa5 0x2d 0xc 0x90 0xc7 0xfc
0x6a 0xa1 0x5f 0x4b 0xdd 0x62 0x39 0x3e 0x55 0x60 0x7f 0xe4 0xf1 0x87
0x26 0xc 0x1f 0xb8 0x7f 0x73 0xa4 0x47 0xdd 0x3a 0x85 0x26 0x4d 0x94 0x7
0x8b 0xd0 0x4e 0xbf 0x3d 0xa0 0xe2 0x8f 0x24 0x66 0x2d]
Another problem caused core dump when i called createVideoWriter while createVideoReader will not core dump:
>>> cv2.cudacodec.createVideoWriter()
terminate called after throwing an instance of 'cv::Exception'
what(): OpenCV(4.5.5)opencv-python/opencv/modules/core/include/opencv2/core/private.cuda.hpp:112: error: (-213:The function/feature is not implemented) The called functionality is disabled for current build or platform in function 'throw_no_cuda'
Thank you. Since your first NAL unit is non IDR I am wondering if the maximum number of unparsed packets is being exceeded. If this was the case I would also expect to see a message similar to the below
[h264 @ 0000027fcbea33c0] co located POCs unavailable
That said if its easy try setting maxUnparsedPackets = 100; and re-building
https://github.com/opencv/opencv_contrib/blob/bbe04482d144f048e798f724eef261dcf4fc8c96/modules/cudacodec/src/video_parser.cpp#L86
Could you also upload and send me the link to local_test.mp4 file and retrieve and write the hex of all frames from ret, encodedFrame = cap.read() to a file and send it to me?
Another problem caused core dump when i called createVideoWriter while createVideoReader will not core dump:
cudacodec::VideoWriter() has not worked for a few years now. If you need to write the output this can be done without encoding it once the python bindings have been fixed.
Could you also upload and send me the link to local_test.mp4 file and retrieve and write the hex of all frames from ret, encodedFrame = cap.read() to a file and send it to me?
I do not know what you need, maybe give me a script.
If its easy try setting
maxUnparsedPackets = 100;and re-building
I will try.
Also my rtsp server is following here . https://github.com/aler9/rtsp-simple-server. Maybe you can setup and test.
I do not know what you need, maybe give me a script.
Try the following then upload out.264 somewhere and send me the link. Then I can run the same bitstream through the decoder at my end and find out what the issue is.
import cv2 as cv
import numpy as np
import sys
np.set_printoptions(formatter={'int':hex})
cap = cv.VideoCapture(url)
cap.set(cv.CAP_PROP_FORMAT,-1)
if(cap.get(cv.CAP_PROP_FORMAT) !=-1): print('Unable to activate raw bitstream reads')
iSpsPps = int(cap.get(cv.CAP_PROP_CODEC_EXTRADATA_INDEX))
ret, spsPps = cap.retrieve(flag=iSpsPps)
if(not ret): print('Unable to retrieve parapeter sets')
f = open("out.264", "wb")
spsPps.tofile(f)
for i in range(100):
ret, encodedFrame = cap.read()
if(not ret):
print('Unable to retrieve encoded frame')
break
encodedFrame.tofile(f)
@cudawarped
If its easy try setting
maxUnparsedPackets = 100;and re-building
It works ! WOW Amazing.
>>> cap.nextFrame()
(True, <cuda_GpuMat 0x7f06c5a84190>)
>>> cap.nextFrame()
(True, <cuda_GpuMat 0x7f06c5a841b0>)
>>> cap.nextFrame()
(True, <cuda_GpuMat 0x7f06c5a841d0>)
>>> cap.nextFrame()
(True, <cuda_GpuMat 0x7f06c5a84170>)
>>> cap.nextFrame()
(True, <cuda_GpuMat 0x7f06c5a84190>)
>>> cap.nextFrame()
(True, <cuda_GpuMat 0x7f06c5a841b0>)
>>> cap.nextFrame()
(True, <cuda_GpuMat 0x7f06c5a841d0>)
So Does this mean that this var maxUnparsedPackets is not set normally?
And another question: If i have a ffmpeg built with cuda, is there anyway to use it such as using VideoCapture with some parameter like ffmepg cmd line.
I'll take a look next week then but it looks like your streaming server is starting at a non IDR frame meaning all the packets until an IDR frame is encountered are discarded exceeding the maximum number of unparsed packets. This could be ffmpeg or your server implementation.
And another question: If i have a ffmpeg built with cuda, is there anyway to use it such as using VideoCapture with some parameter like ffmepg cmd line.
Yes, try
cap = cv.VideoCapture(url, cv.CAP_FFMPEG, [cv.CAP_PROP_HW_ACCELERATION, cv.VIDEO_ACCELERATION_ANY ]);
This is fine for a single stream but will be a lot slower than VideoReader and in my experience will saturate the cpu if you try to decode more than a small number of streams concurrently.
Allright,i will try. But my scene is to use GPU to decode for supporting about 20 video streams at the same time.That is why i must use gpu to decode.
https://docs.opencv.org/4.x/dc/dfc/group__videoio__flags__others.html#gaf61f8388a47aad88cd82cbda6d52c391
Also there is no FLAG for CUDA, maybe you can add it in.
I would be surprised if you can get anywhere near that number without hitting 100% CPU usage. With VideoReader I can easily do 32 1080p h265 streams on GTX 10 series.
With VideoReader I can easily do 32 1080p h265 streams on GTX 10 series
That is what i need. You mean it also hits 100% CPU even using VideoReader ?
No it shouldn't do but this will depend on the CPU. The Nvidia hardware decoder may be near 100% but this will depend on the codec with h265 using less of the decoder than h264.
With VideoReader I can easily do 32 1080p h265 streams on GTX 10 series
Counld you give me some example scripts to do this?
Sorry I don't use python day to day. I would stream in separate threads in c++
Allright , so make VideoReader work again is important. Hah. Let's focus. Now there are two things.
- Set
maxUnparsedPacketscorrectly to make VideoReader work. - Add some features to make VideoCapture can use the ffmpeg with cuda.
Also let's check the dependency. I guess.
- VideoReader need nvidia-video-sdk and ffmpeg(built with or without cuda both ok).
- VideoCapture need ffmpeg built with cuda for this feature.
Are these right?
- Set
maxUnparsedPacketscorrectly to make VideoReader work.
maxUnparsedPackets should work as long as your RTSP source behaves itself and is streamed over TCP. That is, it is a waste for the RTSP server to start output a bitstream at a point (non IDR nal unit) which can't be decoded from. Now if using UDP this could be an issue so maybe the maxUnparsedPackets needs to be altered or user configurable going forward. If you just need to stream from RTSP I would use live555 Media Server, you simply start the application in the folder where your media files are and you can stream any one of them by there file name. The only issue is it only streams elemtentry h264[5] so you would have to extract that from the mp4, but it is much easier than using FFmpeg in combination with an RTSP server.
- Add some features to make VideoCapture can use the ffmpeg with cuda.
Did you try
cap = cv.VideoCapture(url, cv.CAP_FFMPEG, [cv.CAP_PROP_HW_ACCELERATION, cv.VIDEO_ACCELERATION_ANY ]);
I haven't tried this on linux but on windows it uses the Nvidia hardware decoder instead of the CPU. Anyway its sounds like it is unsuitable for you if you need 20 concurrent streams.
VideoReader need nvidia-video-sdk and ffmpeg(built with or without cuda both ok).
To stream from RTSP, it needs nvidia-video-sdk and FFmpeg (built without cuda as this simply provides the bit stream and no decoding).
VideoCapture need ffmpeg built with cuda for this feature.
Yes but as I said above I don't think this will help you due to the number of concurrent streams you want to process. That said I would be interested to find out if you can successfully stream from 20 RTSP sources without saturating the CPU.
Did you try
cap = cv.VideoCapture(url, cv.CAP_FFMPEG, [cv.CAP_PROP_HW_ACCELERATION, cv.VIDEO_ACCELERATION_ANY ]);I haven't tried this on linux but on windows it uses the Nvidia hardware decoder instead of the CPU. Anyway its sounds like it is unsuitable for you if you need 20 concurrent streams.
I try it on Linux, it does not use GPU to decode. So i doubt that
cv.VideoCapture(url, cv.CAP_FFMPEG,v.CAP_PROP_HW_ACCELERATION, cv.VIDEO_ACCELERATION_ANY ]) is the same as cv.VideoCapture(url).
To stream from RTSP, it needs nvidia-video-sdk and FFmpeg (built without cuda as this simply provides the bit stream and no decoding).
Is nvidia-video-sdk the runtime dependency or compiling denpendency or both?
I try to build with cuda and with nvidia-video-sdk get
NVIDIA CUDA: YES (ver 11.4, CUFFT CUBLAS NVCUVID)
NVIDIA GPU arch: 35 37 50 52 60 61 70 75 80 86
NVIDIA PTX archs:
and without nvidia-video-sdk get
NVIDIA CUDA: YES (ver 11.4, CUFFT CUBLAS )
NVIDIA GPU arch: 35 37 50 52 60 61 70 75 80 86
NVIDIA PTX archs:
However,cudacodec can be compiled for both above.
But i found without nvidia-video-sdk in building progress, it can not decode video stream.With an error private.cuda.hpp:112: error: (-213:The function/feature is not implemented) The called functionality is disabled for current build or platform in function 'throw_no_cuda'
Also which nvidia-video-sdk is the real one which opencv need? I have downloaded nvidia-video-sdk and my nvidia driver have the so with the same name.
cp Video_Codec_SDK_11.1.5/Lib/linux/stubs/x86_64/* /usr/local/cuda/lib64
cp Video_Codec_SDK_11.1.5/Interface/* /usr/local/cuda/include
and build opencv then it works.
But i found that Video_Codec_SDK_11.1.5 and my nvidia driver both have the libnvcuvid.so and libnvidia-encode.so.
Same name, different size.
ll Video_Codec_SDK_11.1.5/Interface/
total 340
-rwxrwxrwx 1 root root 67747 Jul 21 2021 cuviddec.h
-rwxrwxrwx 1 root root 246132 Jul 21 2021 nvEncodeAPI.h
-rwxrwxrwx 1 root root 26184 Jul 21 2021 nvcuvid.h
ll Video_Codec_SDK_11.1.5/Lib/linux/stubs/x86_64/
total 8
-rwxrwxrwx 1 root root 3528 Jul 21 2021 libnvcuvid.so
-rwxrwxrwx 1 root root 1480 Jul 21 2021 libnvidia-encode.so
ll /usr/lib64
lrwxrwxrwx 1 root root 21 Apr 16 08:43 libnvidia-encode.so -> libnvidia-encode.so.1
lrwxrwxrwx 1 root root 29 Apr 16 08:43 libnvidia-encode.so.1 -> libnvidia-encode.so.470.57.02
-rwxr-xr-x 1 root root 115K Apr 16 08:43 libnvidia-encode.so.470.57.02
lrwxrwxrwx 1 root root 15 Apr 16 08:43 libnvcuvid.so -> libnvcuvid.so.1
lrwxrwxrwx 1 root root 23 Apr 16 08:43 libnvcuvid.so.1 -> libnvcuvid.so.470.57.02
-rwxr-xr-x 1 root root 5.3M Apr 16 08:43 libnvcuvid.so.470.57.02
It really confused me
Also ffmpeg can compile only with https://github.com/FFmpeg/nv-codec-headers/tree/n11.1.5.1 and without nvidia-video-sdk But opencv need nvidia-video-sdk,nv-codec-headers is not useful for opencv . So what is the differece between them