realtime_object_detection
realtime_object_detection copied to clipboard
gustavz
Does not work with Tensorflow 1.8
I get this error when running source build_tools.py
I know you said your code is supported by Tensorflow 1.4 but I thought it would be good for you to know.
I tested this repo with Tensorflow 1.8 on Jetson TX2 - works fine for me (~20 fps)
Oh, interesting. Do you know why I get that error then? When I run source build_tools.py a new terminal opens up and closes one second later. It shows what you can see above.
I'm not sure what you are trying to do so I will summarize what steps I took in order to make it run: I use Jetson TX2 flashed with Jetpack 3.2 (comes with CUDA 9.0)
- build openCV 3.4.0 (I needed that specific version for other reasons but it should also work with others) - https://www.jetsonhacks.com/2018/05/28/build-opencv-3-4-with-cuda-on-nvidia-jetson-tx2/
- install tensorflow from whl available here: https://devtalk.nvidia.com/default/topic/1031300/tensorflow-1-7-wheel-with-jetpack-3-2-/
- cloned this repo
- run
python3 run_objectdetection.pyfrom the repo dir Works like a charm
Hey there. I'm trying to run the code on my Jetson TX2 using the onboard camera. I have flashed it with Jetpack 3.2, installed OpenCV 3.4 and TensorFlow 1.8 but when I run python run_objectdetection.py (I'm using python 2.7) I get this error:
VIDEOIO ERROR: V4L: device nvcamerasrc ! video/x-raw(memory:NVMM), width=(int)1280, height=(int)720,format=(string)I420, framerate=(fraction)30/1 ! nvvidconv flip-method=0 ! video/x-raw, format=(string)BGRx ! videoconvert ! video/x-raw, format=(string)BGR ! appsink: Unable to query number of channels
[...]
Socket read error. Camera Daemon stopped functioning.....
gst_nvcamera_open() failed ret=0
OpenCV Error: Unspecified error (GStreamer: unable to start pipeline) in icvStartPipeline, file /home/nvidia/opencv/modules/videoio/src/cap_gstreamer.cpp, line 450 [...]
I set my config.yaml file to receive input from the onboard camera by setting the following parameter:
VIDEO_INPUT: "nvcamerasrc ! video/x-raw(memory:NVMM), width=(int)1280, height=(int)720,format=(string)I420, framerate=(fraction)120/1 ! nvvidconv flip-method=0 ! video/x-raw, format=(string)BGRx ! videoconvert ! video/x-raw, format=(string)BGR ! appsink" # Input Must be OpenCV readable
Among others, I have installed with OpenCV using the following 3 flags:
WITH_GSTREAMER=ON -D WITH_FFMPEG=ON -D WITH_LIBV4L=ON -D
Any hint about that error? Thanks
Can you try replacing your VIDEO_INPUT with the following? VIDEO_INPUT: "video_input : nvcamerasrc ! video/x-raw(memory:NVMM), width=(int)600, height=(int)600,format=(string)I420, framerate=(fraction)5/1 ! nvvidconv flip-method=0 ! video/x-raw, format=(string)BGRx ! videoconvert ! video/x-raw, format=(string)BGR ! appsink"
Thanks @xxmarl, unfortunately I get the same error. This is the output:
VIDEOIO ERROR: V4L: device video_input : nvcamerasrc ! video/x-raw(memory:NVMM), width=(int)600, height=(int)600,format=(string)I420, framerate=(fraction)5/1 ! nvvidconv flip-method=0 ! video/x-raw, format=(string)BGRx ! videoconvert ! video/x-raw, format=(string)BGR ! appsink: Unable to query number of channels
Available Sensor modes :
2592 x 1944 FR=30.000000 CF=0x1109208a10 SensorModeType=4 CSIPixelBitDepth=10 DynPixelBitDepth=10
2592 x 1458 FR=30.000000 CF=0x1109208a10 SensorModeType=4 CSIPixelBitDepth=10 DynPixelBitDepth=10
1280 x 720 FR=120.000000 CF=0x1109208a10 SensorModeType=4 CSIPixelBitDepth=10 DynPixelBitDepth=10
NvCameraSrc: Trying To Set Default Camera Resolution. Selected sensorModeIndex = 1 WxH = 2592x1458 FrameRate = 30.000000 ...
Socket read error. Camera Daemon stopped functioning.....
gst_nvcamera_open() failed ret=0
OpenCV Error: Unspecified error (GStreamer: unable to start pipeline
) in icvStartPipeline, file /home/nvidia/opencv/modules/videoio/src/cap_gstreamer.cpp, line 450
Traceback (most recent call last):
File "run_objectdetection.py", line 180, in <module>
detection(model, config)
File "run_objectdetection.py", line 35, in detection
vs = WebcamVideoStream(config.VIDEO_INPUT,config.WIDTH,config.HEIGHT).start()
File "/home/nvidia/project/realtime_object_detection/rod/helper.py", line 177, in __init__
self.stream.set(cv2.CAP_PROP_FRAME_WIDTH, self.width)
cv2.error: /home/nvidia/opencv/modules/videoio/src/cap_gstreamer.cpp:450: error: (-2) GStreamer: unable to start pipeline
in function icvStartPipeline
This is the output of cv2.getBuildInformation().
print(cv2.getBuildInformation())
General configuration for OpenCV 3.4.0 =====================================
Version control: unknown
Platform:
Timestamp: 2018-06-15T12:19:00Z
Host: Linux 4.4.38-tegra aarch64
CMake: 3.5.1
CMake generator: Unix Makefiles
CMake build tool: /usr/bin/make
Configuration: RELEASE
CPU/HW features:
Baseline: NEON FP16
required: NEON
disabled: VFPV3
C/C++:
Built as dynamic libs?: YES
C++ Compiler: /usr/bin/c++ (ver 5.4.0)
C++ flags (Release): -fsigned-char -W -Wall -Werror=return-type -Werror=non-virtual-dtor -Werror=address -Werror=sequence-point -Wformat -Werror=format-security -Wmissing-declarations -Wundef -Winit-self -Wpointer-arith -Wshadow -Wsign-promo -Wuninitialized -Winit-self -Wno-narrowing -Wno-delete-non-virtual-dtor -Wno-comment -fdiagnostics-show-option -pthread -fomit-frame-pointer -ffunction-sections -fdata-sections -fvisibility=hidden -fvisibility-inlines-hidden -O3 -DNDEBUG -DNDEBUG
C++ flags (Debug): -fsigned-char -W -Wall -Werror=return-type -Werror=non-virtual-dtor -Werror=address -Werror=sequence-point -Wformat -Werror=format-security -Wmissing-declarations -Wundef -Winit-self -Wpointer-arith -Wshadow -Wsign-promo -Wuninitialized -Winit-self -Wno-narrowing -Wno-delete-non-virtual-dtor -Wno-comment -fdiagnostics-show-option -pthread -fomit-frame-pointer -ffunction-sections -fdata-sections -fvisibility=hidden -fvisibility-inlines-hidden -g -O0 -DDEBUG -D_DEBUG
C Compiler: /usr/bin/cc
C flags (Release): -fsigned-char -W -Wall -Werror=return-type -Werror=non-virtual-dtor -Werror=address -Werror=sequence-point -Wformat -Werror=format-security -Wmissing-declarations -Wmissing-prototypes -Wstrict-prototypes -Wundef -Winit-self -Wpointer-arith -Wshadow -Wuninitialized -Winit-self -Wno-narrowing -Wno-comment -fdiagnostics-show-option -pthread -fomit-frame-pointer -ffunction-sections -fdata-sections -fvisibility=hidden -O3 -DNDEBUG -DNDEBUG
C flags (Debug): -fsigned-char -W -Wall -Werror=return-type -Werror=non-virtual-dtor -Werror=address -Werror=sequence-point -Wformat -Werror=format-security -Wmissing-declarations -Wmissing-prototypes -Wstrict-prototypes -Wundef -Winit-self -Wpointer-arith -Wshadow -Wuninitialized -Winit-self -Wno-narrowing -Wno-comment -fdiagnostics-show-option -pthread -fomit-frame-pointer -ffunction-sections -fdata-sections -fvisibility=hidden -g -O0 -DDEBUG -D_DEBUG
Linker flags (Release):
Linker flags (Debug):
ccache: NO
Precompiled headers: YES
Extra dependencies: dl m pthread rt cudart nppc nppial nppicc nppicom nppidei nppif nppig nppim nppist nppisu nppitc npps cublas cufft -L/usr/local/cuda-9.0/lib64
3rdparty dependencies:
OpenCV modules:
To be built: calib3d core cudaarithm cudabgsegm cudacodec cudafeatures2d cudafilters cudaimgproc cudalegacy cudaobjdetect cudaoptflow cudastereo cudawarping cudev dnn features2d flann highgui imgcodecs imgproc ml objdetect photo python2 python_bindings_generator shape stitching superres video videoio videostab
Disabled: java js world
Disabled by dependency: -
Unavailable: python3 ts viz
Applications: apps
Documentation: NO
Non-free algorithms: NO
GUI:
GTK+: YES (ver 2.24.30)
GThread : YES (ver 2.48.2)
GtkGlExt: NO
VTK support: NO
Media I/O:
ZLib: /usr/lib/aarch64-linux-gnu/libz.so (ver 1.2.8)
JPEG: /usr/lib/aarch64-linux-gnu/libjpeg.so (ver )
WEBP: build (ver encoder: 0x020e)
PNG: /usr/lib/aarch64-linux-gnu/libpng.so (ver 1.2.54)
TIFF: /usr/lib/aarch64-linux-gnu/libtiff.so (ver 42 / 4.0.6)
JPEG 2000: /usr/lib/aarch64-linux-gnu/libjasper.so (ver 1.900.1)
OpenEXR: build (ver 1.7.1)
Video I/O:
DC1394: YES (ver 2.2.4)
FFMPEG: YES
avcodec: YES (ver 56.60.100)
avformat: YES (ver 56.40.101)
avutil: YES (ver 54.31.100)
swscale: YES (ver 3.1.101)
avresample: NO
GStreamer:
base: YES (ver 1.8.3)
video: YES (ver 1.8.3)
app: YES (ver 1.8.3)
riff: YES (ver 1.8.3)
pbutils: YES (ver 1.8.3)
libv4l/libv4l2: 1.10.0 / 1.10.0
v4l/v4l2: linux/videodev2.h
gPhoto2: NO
Parallel framework: pthreads
Trace: YES (built-in)
Other third-party libraries:
Lapack: NO
Eigen: NO
Custom HAL: YES (carotene (ver 0.0.1))
NVIDIA CUDA: YES (ver 9.0, CUFFT CUBLAS)
NVIDIA GPU arch: 62
NVIDIA PTX archs:
Python 2:
Interpreter: /usr/bin/python2.7 (ver 2.7.12)
Libraries: /usr/lib/aarch64-linux-gnu/libpython2.7.so (ver 2.7.12)
numpy: /usr/local/lib/python2.7/dist-packages/numpy/core/include (ver 1.14.3)
packages path: lib/python2.7/dist-packages
Python (for build): /usr/bin/python2.7
Java:
ant: /usr/bin/ant (ver 1.9.6)
JNI: /usr/lib/jvm/java-8-oracle/include /usr/lib/jvm/java-8-oracle/include/linux /usr/lib/jvm/java-8-oracle/include
Java wrappers: NO
Java tests: NO
Matlab: NO
Install to: /usr/local
In the Video I/O section GStreamer seems to be installed correctly.
I could run it successfully on my laptop which has OpenCV 3.4, Tensorflow 1.4.0 and an NVIDIA GeForce 950M (it ran at about 3/4 FPS and I was using the laptop's camera).
Any idea on how to make it run on the TX2 given the information above?
Hi @BourbonCreams,
=== About laptop === Tensorflow 1.4.0 doesn't support CUDA9. Perhaps your Tensorflow is running on cpu only. (and build without cpu architecture optimization.)
try:
pip install tensorflow-gpu
on your ubuntu laptop.
=== About TX2 ===
try:
sudo apt-get install libxine2
check onboard cam with code:
import cv2
src = "nvcamerasrc ! video/x-raw(memory:NVMM), width=(int)1280, height=(int)720,format=(string)I420, framerate=(fraction)30/1 ! nvvidconv flip-method=0 ! video/x-raw, format=(string)BGRx ! videoconvert ! video/x-raw, format=(string)BGR ! appsink"
vid = cv2.VideoCapture(src)
if not vid.isOpened():
raise IOError(("Couldn't open video file or webcam."))
print("Press 'q' to Exit")
while True:
retval, cv_bgr = vid.read()
if not retval:
break
cv2.imshow('camera', cv_bgr)
if cv2.waitKey(1) & 0xFF == ord('q'):
break
vid.release()
config.yml
### USB Webcam on PC ###
#video_input: 0
### USB Webcam on TX2 ###
#video_input: 1
### Onboard camera on TX2 ###
video_input: "nvcamerasrc ! video/x-raw(memory:NVMM), width=(int)1280, height=(int)720,format=(string)I420, framerate=(fraction)30/1 ! nvvidconv flip-method=0 ! video/x-raw, format=(string)BGRx ! videoconvert ! video/x-raw, format=(string)BGR ! appsink"
Hi @naisy, thank you for your support.
=== About laptop === I am not too concerned about my laptop, I would need this model to work on the Jetson so I'm not worried about using my GPU with Tensorflow (you're right, I have it installed without GPU support).
=== About TX2 ===
I installed the library you told me to install and a re-installed OpenCV with the flag -D WITH_XINE=ON but it still does not work. This is the output I get from python run_objectdetection.py:
VIDEOIO ERROR: V4L: device nvcamerasrc ! video/x-raw(memory:NVMM), width=(int)1280, height=(int)720,format=(string)I420, framerate=(fraction)30/1 ! nvvidconv flip-method=0 ! video/x-raw, format=(string)BGRx ! videoconvert ! video/x-raw, format=(string)BGR ! appsink: Unable to query number of channels
(ERROR)icvOpenAVI_XINE(): Unable to open source 'nvcamerasrc ! video/x-raw(memory:NVMM), width=(int)1280, height=(int)720,format=(string)I420, framerate=(fraction)30/1 ! nvvidconv flip-method=0 ! video/x-raw, format=(string)BGRx ! videoconvert ! video/x-raw, format=(string)BGR ! appsink'
Available Sensor modes :
2592 x 1944 FR=30.000000 CF=0x1109208a10 SensorModeType=4 CSIPixelBitDepth=10 DynPixelBitDepth=10
2592 x 1458 FR=30.000000 CF=0x1109208a10 SensorModeType=4 CSIPixelBitDepth=10 DynPixelBitDepth=10
1280 x 720 FR=120.000000 CF=0x1109208a10 SensorModeType=4 CSIPixelBitDepth=10 DynPixelBitDepth=10
NvCameraSrc: Trying To Set Default Camera Resolution. Selected sensorModeIndex = 1 WxH = 2592x1458 FrameRate = 30.000000 ...
Socket read error. Camera Daemon stopped functioning.....
gst_nvcamera_open() failed ret=0
OpenCV Error: Unspecified error (GStreamer: unable to start pipeline
) in icvStartPipeline, file /home/nvidia/opencv/modules/videoio/src/cap_gstreamer.cpp, line 450
Traceback (most recent call last):
File "run_objectdetection.py", line 180, in <module>
detection(model, config)
File "run_objectdetection.py", line 35, in detection
vs = WebcamVideoStream(config.VIDEO_INPUT,config.WIDTH,config.HEIGHT).start()
File "/home/nvidia/project/realtime_object_detection/rod/helper.py", line 177, in __init__
self.stream.set(cv2.CAP_PROP_FRAME_WIDTH, self.width)
cv2.error: /home/nvidia/opencv/modules/videoio/src/cap_gstreamer.cpp:450: error: (-2) GStreamer: unable to start pipeline
in function icvStartPipeline
After running your script the video stream works but on Terminal I still see this cheeky output:
nvidia@tegra-ubuntu:~$ python cameraTrial.py
VIDEOIO ERROR: V4L: device nvcamerasrc ! video/x-raw(memory:NVMM), width=(int)1280, height=(int)720,format=(string)I420, framerate=(fraction)30/1 ! nvvidconv flip-method=0 ! video/x-raw, format=(string)BGRx ! videoconvert ! video/x-raw, format=(string)BGR ! appsink: Unable to query number of channels
(ERROR)icvOpenAVI_XINE(): Unable to open source 'nvcamerasrc ! video/x-raw(memory:NVMM), width=(int)1280, height=(int)720,format=(string)I420, framerate=(fraction)30/1 ! nvvidconv flip-method=0 ! video/x-raw, format=(string)BGRx ! videoconvert ! video/x-raw, format=(string)BGR ! appsink'
Available Sensor modes :
2592 x 1944 FR=30.000000 CF=0x1109208a10 SensorModeType=4 CSIPixelBitDepth=10 DynPixelBitDepth=10
2592 x 1458 FR=30.000000 CF=0x1109208a10 SensorModeType=4 CSIPixelBitDepth=10 DynPixelBitDepth=10
1280 x 720 FR=120.000000 CF=0x1109208a10 SensorModeType=4 CSIPixelBitDepth=10 DynPixelBitDepth=10
NvCameraSrc: Trying To Set Default Camera Resolution. Selected sensorModeIndex = 1 WxH = 2592x1458 FrameRate = 30.000000 ...
Press 'q' to Exit
The good news is that if I plug in an external webcam and set VIDEO_INPUT:1 in the config.yml file then everything works (I get around 27 fps, which is good). I'm not sure why the onboard camera does not want to work. I have run other algorithms such as Faster R-CNN and YOLO from DarkNet using the onboard camera. For YOLO I also needed to specify the camera with this parameter:
"nvcamerasrc ! video/x-raw(memory:NVMM), width=(int)1280, height=(int)720,format=(string)I420, framerate=(fraction)30/1 ! nvvidconv flip-method=0 ! video/x-raw, format=(string)BGRx ! videoconvert ! video/x-raw, format=(string)BGR ! appsink"
but it won't work for SSD_with_MobileNet. I am happy to run this with the webcam, I don't particularly need the onboard camera, but it would good for future reference to solve this issue so if guys have any other idea I would gladly help you solve this problem with your guidance.
Thanks, Lorenzo
Hi @BourbonCreams,
This is my log. Something error, but onboard camera worked. python a
(ERROR)icvOpenAVI_XINE(): Unable to initialize video driver.
Available Sensor modes :
2592 x 1944 FR=30.000000 CF=0x1109208a10 SensorModeType=4 CSIPixelBitDepth=10 DynPixelBitDepth=10
2592 x 1458 FR=30.000000 CF=0x1109208a10 SensorModeType=4 CSIPixelBitDepth=10 DynPixelBitDepth=10
1280 x 720 FR=120.000000 CF=0x1109208a10 SensorModeType=4 CSIPixelBitDepth=10 DynPixelBitDepth=10
NvCameraSrc: Trying To Set Default Camera Resolution. Selected sensorModeIndex = 1 WxH = 2592x1458 FrameRate = 30.000000 ...
Press 'q' to Exit
Could not initialize OpenGL for RasterGLSurface, reverting to RasterSurface.
Maybe, you need to install something package. I always install these packages for build OpenCV, Tensorflow and pip on TX2.
apt-get install -y python-dev libjpeg-dev libxslt-dev libxml2-dev libffi-dev libcurl4-openssl-dev libssl-dev libblas-dev liblapack-dev gfortran libpng12-dev libfreetype6-dev
apt-get install -y zip curl locate libprotobuf-dev build-essential cmake libeigen3-dev libatlas-base-dev gfortran wget libavformat-dev libavcodec-dev libswscale-dev libavresample-dev ffmpeg pkg-config unzip qtbase5-dev libgtk-3-dev libdc1394-22 libdc1394-22-dev libjpeg-dev libpng12-dev libtiff5-dev libjasper-dev libavcodec-dev libavformat-dev libswscale-dev libxine2-dev libgstreamer1.0-dev libgstreamer-plugins-base1.0-dev libv4l-dev libtbb-dev libfaac-dev libmp3lame-dev libopencore-amrnb-dev libopencore-amrwb-dev libtheora-dev libvorbis-dev libxvidcore-dev v4l-utils liblapacke-dev libopenblas-dev checkinstall libgdal-dev libgphoto2-dev
I have just finished building OpenCV 3.4.1(JetPack3.2.1/Python2.7) which enabled the Tracking API.
Onboard camera worked with this package. (Even JetPack 3.2 should work)
https://github.com/naisy/JetsonTX2/tree/JetPack3.2_python2.7/JetPack3.2/python2.7/binary/
dpkg -i opencv-3.4.1.deb
