realtime_object_detection icon indicating copy to clipboard operation
realtime_object_detection copied to clipboard

Published 20 hours ago • verified publisher icon gustavz

jetson TX2 camera

Open imxboards opened this issue 7 years ago • 13 comments

hi , I install tx2 with Ubuntu 16.04 Python 2.7 Tensorflow 1.7 OpenCV 3.3.1 But when I ruan object_detection.py issue is coming

vidia@tegra-ubuntu:~/tf/realtime_object_detection$ python object_detection.py

Model found. Proceed. Loading frozen model into memory 2018-03-30 07:11:08.517989: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:865] ARM64 does not support NUMA - returning NUMA node zero 2018-03-30 07:11:08.518123: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1344] Found device 0 with properties: name: NVIDIA Tegra X2 major: 6 minor: 2 memoryClockRate(GHz): 1.3005 pciBusID: 0000:00:00.0 totalMemory: 7.66GiB freeMemory: 2.29GiB 2018-03-30 07:11:08.518172: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1423] Adding visible gpu devices: 0 2018-03-30 07:11:10.014011: I tensorflow/core/common_runtime/gpu/gpu_device.cc:911] Device interconnect StreamExecutor with strength 1 edge matrix: 2018-03-30 07:11:10.014083: I tensorflow/core/common_runtime/gpu/gpu_device.cc:917] 0 2018-03-30 07:11:10.014109: I tensorflow/core/common_runtime/gpu/gpu_device.cc:930] 0: N 2018-03-30 07:11:10.014269: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1041] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 1694 MB memory) -> physical GPU (device: 0, name: NVIDIA Tegra X2, pci bus id: 0000:00:00.0, compute capability: 6.2) Loading label map Building Graph 2018-03-30 07:11:33.197609: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1423] Adding visible gpu devices: 0 2018-03-30 07:11:33.197707: I tensorflow/core/common_runtime/gpu/gpu_device.cc:911] Device interconnect StreamExecutor with strength 1 edge matrix: 2018-03-30 07:11:33.197737: I tensorflow/core/common_runtime/gpu/gpu_device.cc:917] 0 2018-03-30 07:11:33.197758: I tensorflow/core/common_runtime/gpu/gpu_device.cc:930] 0: N 2018-03-30 07:11:33.197847: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1041] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 1694 MB memory) -> physical GPU (device: 0, name: NVIDIA Tegra X2, pci bus id: 0000:00:00.0, compute capability: 6.2) start!!! 2018-03-30 07:11:33.199546: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1423] Adding visible gpu devices: 0 2018-03-30 07:11:33.199636: I tensorflow/core/common_runtime/gpu/gpu_device.cc:911] Device interconnect StreamExecutor with strength 1 edge matrix: 2018-03-30 07:11:33.199671: I tensorflow/core/common_runtime/gpu/gpu_device.cc:917] 0 2018-03-30 07:11:33.199713: I tensorflow/core/common_runtime/gpu/gpu_device.cc:930] 0: N 2018-03-30 07:11:33.199817: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1041] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 1694 MB memory) -> physical GPU (device: 0, name: NVIDIA Tegra X2, pci bus id: 0000:00:00.0, compute capability: 6.2) 2018-03-30 07:11:33.200175: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1423] Adding visible gpu devices: 0 2018-03-30 07:11:33.200259: I tensorflow/core/common_runtime/gpu/gpu_device.cc:911] Device interconnect StreamExecutor with strength 1 edge matrix: 2018-03-30 07:11:33.200292: I tensorflow/core/common_runtime/gpu/gpu_device.cc:917] 0 2018-03-30 07:11:33.200315: I tensorflow/core/common_runtime/gpu/gpu_device.cc:930] 0: N 2018-03-30 07:11:33.200394: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1041] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 1694 MB memory) -> physical GPU (device: 0, name: NVIDIA Tegra X2, pci bus id: 0000:00:00.0, compute capability: 6.2) VIDEOIO ERROR: V4L2: Pixel format of incoming image is unsupported by OpenCV OpenCV Error: Unspecified error (GStreamer: unable to start pipeline ) in cvCaptureFromCAM_GStreamer, file /home/nvidia/opencv/modules/videoio/src/cap_gstreamer.cpp, line 887 VIDEOIO(cvCreateCapture_GStreamer(CV_CAP_GSTREAMER_V4L2, reinterpret_cast<char *>(index))): raised OpenCV exception:

/home/nvidia/opencv/modules/videoio/src/cap_gstreamer.cpp:887: error: (-2) GStreamer: unable to start pipeline in function cvCaptureFromCAM_GStreamer

Start video stream with shape: 0,0 Press 'q' to Exit Starting Detection OpenCV Error: Assertion failed (scn == 3 || scn == 4) in cvtColor, file /home/nvidia/opencv/modules/imgproc/src/color.cpp, line 11016 Traceback (most recent call last): File "object_detection.py", line 302, in main() File "object_detection.py", line 298, in main detection(graph, category, score, expand) File "object_detection.py", line 213, in detection image_expanded = np.expand_dims(cv2.cvtColor(image, cv2.COLOR_BGR2RGB), axis=0) cv2.error: /home/nvidia/opencv/modules/imgproc/src/color.cpp:11016: error: (-215) scn == 3 || scn == 4 in function cvtColor

can you help me ? should i change config.yml parm?

imxboards avatar Mar 30 '18 07:03 imxboards

are you using the onboard camera of the jetson? do you know how to use it in opencv, have a look here: https://devtalk.nvidia.com/default/topic/1017860/using-on-board-tx2-camera-with-opencv-3-2-/?offset=7 and others

gustavz avatar Apr 02 '18 11:04 gustavz

Hi! for my experience: In config.sample.yml

video_input : nvcamerasrc ! video/x-raw(memory:NVMM), width=(int)600, height=(int)600,format=(string)I420, framerate=(fraction)5/1 ! nvvidconv flip-method=0 ! video/x-raw, format=(string)BGRx ! videoconvert ! video/x-raw, format=(string)BGR ! appsink

Alro10 avatar Apr 14 '18 15:04 Alro10

@AlexanderRobles21 @GustavZ Hi, I tried setting video_input as you suggested for Jetson TX2 camera, however, I get the following error.

nvidia@tegra-ubuntu:~/Desktop/realtime_object_detection$ python object_detection.py 
> Model found. Proceed.
> Loading frozen model into memory
2018-04-26 22:56:07.466932: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:865] ARM64 does not support NUMA - returning NUMA node zero
2018-04-26 22:56:07.467125: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1212] Found device 0 with properties: 
name: NVIDIA Tegra X2 major: 6 minor: 2 memoryClockRate(GHz): 1.3005
pciBusID: 0000:00:00.0
totalMemory: 7.66GiB freeMemory: 2.29GiB
2018-04-26 22:56:07.467194: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1312] Adding visible gpu devices: 0
2018-04-26 22:56:10.025344: I tensorflow/core/common_runtime/gpu/gpu_device.cc:993] Creating TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 1687 MB memory) -> physical GPU (device: 0, name: NVIDIA Tegra X2, pci bus id: 0000:00:00.0, compute capability: 6.2)
> Loading label map
> Building Graph
2018-04-26 22:56:32.894441: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1312] Adding visible gpu devices: 0
2018-04-26 22:56:32.894590: I tensorflow/core/common_runtime/gpu/gpu_device.cc:993] Creating TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 577 MB memory) -> physical GPU (device: 0, name: NVIDIA Tegra X2, pci bus id: 0000:00:00.0, compute capability: 6.2)
VIDEOIO ERROR: V4L: device nvcamerasrc ! video/x-raw(memory:NVMM), width=(int)600, height=(int)600,format=(string)I420, framerate=(fraction)5/1 ! nvvidconv flip-method=0 ! video/x-raw, format=(string)BGRx ! videoconvert ! video/x-raw, format=(string)BGR ! appsink: Unable to query number of channels
2018-04-26 22:56:32.897282: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1312] Adding visible gpu devices: 0
2018-04-26 22:56:32.897409: I tensorflow/core/common_runtime/gpu/gpu_device.cc:993] Creating TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 577 MB memory) -> physical GPU (device: 0, name: NVIDIA Tegra X2, pci bus id: 0000:00:00.0, compute capability: 6.2)
2018-04-26 22:56:32.902302: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1312] Adding visible gpu devices: 0
2018-04-26 22:56:32.902475: I tensorflow/core/common_runtime/gpu/gpu_device.cc:993] Creating TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 577 MB memory) -> physical GPU (device: 0, name: NVIDIA Tegra X2, pci bus id: 0000:00:00.0, compute capability: 6.2)

Available Sensor modes : 
2592 x 1944 FR=30.000000 CF=0x1109208a10 SensorModeType=4 CSIPixelBitDepth=10 DynPixelBitDepth=10
2592 x 1458 FR=30.000000 CF=0x1109208a10 SensorModeType=4 CSIPixelBitDepth=10 DynPixelBitDepth=10
1280 x 720 FR=120.000000 CF=0x1109208a10 SensorModeType=4 CSIPixelBitDepth=10 DynPixelBitDepth=10

NvCameraSrc: Trying To Set Default Camera Resolution. Selected sensorModeIndex = 1 WxH = 2592x1458 FrameRate = 30.000000 ...

Socket read error. Camera Daemon stopped functioning.....
gst_nvcamera_open() failed ret=0
OpenCV Error: Unspecified error (GStreamer: unable to start pipeline
) in icvStartPipeline, file /home/nvidia/src/opencv-3.4.0/modules/videoio/src/cap_gstreamer.cpp, line 450
Traceback (most recent call last):
  File "object_detection.py", line 301, in <module>
    main()
  File "object_detection.py", line 297, in main
    detection(graph, category, score, expand)
  File "object_detection.py", line 201, in detection
    video_stream = WebcamVideoStream(video_input,width,height).start()
  File "/home/nvidia/Desktop/realtime_object_detection/stuff/helper.py", line 108, in __init__
    self.stream.set(cv2.CAP_PROP_FRAME_WIDTH, self.width)
cv2.error: /home/nvidia/src/opencv-3.4.0/modules/videoio/src/cap_gstreamer.cpp:450: error: (-2) GStreamer: unable to start pipeline
 in function icvStartPipeline

I have installed applicable gstreamer dev packages. I am able to invoke the camera through terminal via gst-launch, for example gst-launch-1.0 -ev nvcamerasrc ! nvvidconv ! ximagesink and can also access camera via other scripts for example

  # Use Jetson onboard camera
    gst_str = ("nvcamerasrc ! "
               "video/x-raw(memory:NVMM), width=(int)2592, height=(int)1458, format=(string)I420, framerate=(fraction)30/1 ! "
               "nvvidconv ! video/x-raw, width=(int){}, height=(int){}, format=(string)BGRx ! "
               "videoconvert ! appsink").format(width, height)
    return cv2.VideoCapture(gst_str, cv2.CAP_GSTREAMER)

Any help would be appreciated. Thanks.

warisz avatar Apr 27 '18 06:04 warisz

I make a guess: you are using OpenCV that comes preinstalled with flashing JetPack right?

The problem is that Nvidia/JetPack Build OpenCV without Gstreamer support. That’s why it is not possible to use the onboard camera with the standard OpenCV (ask NVIDIA why they install OpenCV that way that it can’t use its own hardware. They also disabled CUDA support for OpenCV, so i guess they dont want you to use it but instead Visionworks...)

Long story short: you need to completely deinstall OpenCV and reinstall it from source with Gstreamer Support Enabled.

gustavz avatar Apr 27 '18 06:04 gustavz

Hi Gustav,

Thanks for your quick reply. I actually reinstalled OpenCV from source following this tutorial: https://jkjung-avt.github.io/opencv3-on-tx2/ I am able to launch the onboard camera, like for example, via the last code snippet I posted in my previous message.

I am using TF1.6 with JetPack 3.1 (and have also tried TF1.5 and gotten same error). From prying around the other issues in this repo, TF1.4 is recommended for this repo; however the error I have posted seems to be Gstreamer related.

I will try reinstalling OpenCV from source and switch to JetPack3.1 to use TF1.4 and try that and post an update. Any input from your side is also appreciated.

Thanks a lot.

warisz avatar Apr 27 '18 06:04 warisz

I use and always have used JetPack 3.2, maybe you should switch to that?

gustavz avatar Apr 27 '18 08:04 gustavz

Hi @imxboards,

I recently tested object_detection.py for Jetpack 3.1 and TF 1.3 without any problems using onboard cam and webcam ( video_input = /dev/video1) achieved 20fps. But when I used TF 1.6 rc1 there is a problem:

2018-04-27 11:06:05.483867: E tensorflow/stream_executor/event.cc:33] error destroying CUDA event in context 0x337fbf0: CUDA_ERROR_LAUNCH_FAILED

I recommend to use TF 1.3, I think this version works for CUDA 8. For Jetpack 3.2 (CUDA 9) I didn't tested yet.

Alro10 avatar Apr 27 '18 14:04 Alro10

I successfully tested it on 1.4, 1.5, 1.6 but with varying performance results.

If you want to achieve highest results use the versions that I recommend in the readme.

Also I strongly recommend you all to use JetPack 3.2 because of CUDA 9 / cuDNN 7 among other improvements

gustavz avatar Apr 27 '18 14:04 gustavz

@warisz could you solve the problem? I am experiencing the same problem trying to install with tensorflow 1.6 and opencv 3.4.0 that i built by myself... cheers!

lfelipesv avatar May 14 '18 07:05 lfelipesv

I am experiencing the same problem, I have Jetpack 3.3 and TF 1.9

I installed open CV using this tutorial : https://jkjung-avt.github.io/opencv3-on-tx2/

VIDEOIO ERROR: V4L: device nvcamerasrc ! video/x-raw(memory:NVMM), width=(int)1280, height=(int)720,format=(string)I420, framerate=(fraction)30/1 ! nvvidconv flip-method=0 ! video/x-raw, format=(string)BGRx ! videoconvert ! video/x-raw, format=(string)BGR ! appsink: Unable to query number of channels

Available Sensor modes : 2592 x 1944 FR=30.000000 CF=0x1109208a10 SensorModeType=4 CSIPixelBitDepth=10 DynPixelBitDepth=10 2592 x 1458 FR=30.000000 CF=0x1109208a10 SensorModeType=4 CSIPixelBitDepth=10 DynPixelBitDepth=10 1280 x 720 FR=120.000000 CF=0x1109208a10 SensorModeType=4 CSIPixelBitDepth=10 DynPixelBitDepth=10

NvCameraSrc: Trying To Set Default Camera Resolution. Selected sensorModeIndex = 1 WxH = 2592x1458 FrameRate = 30.000000 ...

Available Sensor modes : 2592 x 1944 FR=30.000000 CF=0x1109208a10 SensorModeType=4 CSIPixelBitDepth=10 DynPixelBitDepth=10 2592 x 1458 FR=30.000000 CF=0x1109208a10 SensorModeType=4 CSIPixelBitDepth=10 DynPixelBitDepth=10 1280 x 720 FR=120.000000 CF=0x1109208a10 SensorModeType=4 CSIPixelBitDepth=10 DynPixelBitDepth=10

NvCameraSrc: Trying To Set Default Camera Resolution. Selected sensorModeIndex = 1 WxH = 2592x1458 FrameRate = 30.000000 ...

Socket read error. Camera Daemon stopped functioning..... OpenCV Error: Unspecified error (GStreamer: unable to start pipeline ) in icvStartPipeline, file /home/nvidia/src/opencv-3.4.0/modules/videoio/src/cap_gstreamer.cpp, line 450 Traceback (most recent call last): File "run_stream.py", line 123, in main detection.start(cfg) File "/home/nvidia/Desktop/realtime_object_detection-master/lib/detection_nms_v2.py", line 192, in start video_reader.start(VIDEO_INPUT, WIDTH, HEIGHT, save_to_file=SAVE_TO_FILE) File "/home/nvidia/Desktop/realtime_object_detection-master/lib/webcam.py", line 45, in start self.vid.set(cv2.CAP_PROP_FRAME_HEIGHT, height) cv2.error: /home/nvidia/src/opencv-3.4.0/modules/videoio/src/cap_gstreamer.cpp:450: error: (-2) GStreamer: unable to start pipeline in function icvStartPipeline

scbean avatar Sep 17 '18 05:09 scbean

Hi @scbean have similar problem to the one you are describing did you find a solution for it ???

Merwan

Merwanski avatar Feb 13 '19 16:02 Merwanski

Hi @Merwanski and @scbean

You can check my repository: https://github.com/Alro10/realtime_object_detection

Use this file: object_detectionjetson.py

Please write back if that solves your problem!

Alro10 avatar Feb 14 '19 15:02 Alro10

Thank you @Alro10 ... the camera is working but I am still getting this error message """""" VIDEOIO ERROR: V4L: device nvcamerasrc ! video/x-raw(memory:NVMM), width=(int)1280, height=(int)720,format=(string)I420, framerate=(fraction)30/1 ! nvvidconv flip-method=0 ! video/x-raw, format=(string)BGRx ! videoconvert ! video/x-raw, format=(string)BGR ! appsink: Unable to query number of channels """""""

Merwanski avatar Feb 15 '19 21:02 Merwanski