OrbbecSDK icon indicating copy to clipboard operation
OrbbecSDK copied to clipboard

Femto Bolt: Crash after "got nvram data failed.retrying..."

Open puzzlepaint opened this issue 1 year ago • 7 comments

Hi, with a Femto Bolt device I am getting an issue that it sometimes fails to open. The issue can be reproduced with OrbbecViewer as follows:

  • Open OrbbecViewer with the device connected
  • Start the color and depth streams
  • Close the OrbbecViewer window
  • Repeat these three steps until the issue occurs

Roughly every 3rd time, OrbbecViewer will fail to open, with the following log output:

2024-03-23 13:14:22.156 INFO  [11368] [loggerInit@18] **********************************************
2024-03-23 13:14:22.156 INFO  [11368] [loggerInit@19]  OrbbecViewer launched! Welcome!! 
2024-03-23 13:14:22.156 INFO  [11368] [loggerInit@20]  	- Version: V1.9.5
2024-03-23 13:14:22.156 INFO  [11368] [loggerInit@21]  	- Author: 
2024-03-23 13:14:22.156 INFO  [11368] [loggerInit@22]  	- E-Mail: 
2024-03-23 13:14:22.156 INFO  [11368] [loggerInit@23]  	- Company: orbbec
2024-03-23 13:14:22.156 INFO  [11368] [loggerInit@24]  	- Website: http://www.orbbec.com.cn/
2024-03-23 13:14:22.156 INFO  [11368] [loggerInit@25] **********************************************
[03/23 13:14:22.285349][info][11368][Context.cpp:69] Context created with config: /home/thomas/Downloads/OrbbecViewer_v1.9.5_202403050503_linux_x64_release/OrbbecSDKConfig_v1.0.xml
[03/23 13:14:22.285362][info][11368][Context.cpp:74] Context work_dir=/home/thomas/Downloads/OrbbecViewer_v1.9.5_202403050503_linux_x64_release
[03/23 13:14:22.285368][info][11368][Context.cpp:77] 	- SDK version: 1.9.5
[03/23 13:14:22.285373][info][11368][Context.cpp:78] 	- SDK stage version: main
[03/23 13:14:22.285381][info][11368][Context.cpp:82] get config EnumerateNetDevice:false
[03/23 13:14:22.285390][info][11368][LinuxPal.cpp:38] createObPal: create LinuxPal!
[03/23 13:14:22.383854][info][11368][LinuxPal.cpp:112] Create PollingDeviceWatcher!
[03/23 13:14:22.383884][info][11368][DeviceManager.cpp:15] Current found device(s): (1)
[03/23 13:14:22.383890][info][11368][DeviceManager.cpp:24] 	- Name: Femto Bolt, PID: 0x066b, SN/ID: CL8FC3100E2, Connection: USB3.1
[03/23 13:14:22.383916][info][11368][DeviceManager.cpp:310] Enable net device enumeration: true
[03/23 13:14:22.384039][info][11368][GVCPClient.cpp:223] bind 0.0.0.0:0
[03/23 13:14:23.385369][info][11368][DeviceManager.cpp:15] Current device(s) list: (1)
[03/23 13:14:23.385388][info][11368][DeviceManager.cpp:24] 	- Name: Femto Bolt, PID: 0x066b, SN/ID: CL8FC3100E2, Connection: USB3.1
[03/23 13:14:23.385451][info][11368][FemtoBoltUvcDevice.cpp:23] FemtoBoltUvcDevice init ...
[03/23 13:14:23.385505][info][11368][FemtoBoltUvcDevice.cpp:120] Create command start!
[03/23 13:14:23.386569][info][11368][MSDEConverterDevice.cpp:721] Succeed to load depth engine plugin
[03/23 13:14:23.592557][info][11368][FemtoBoltUvcDevice.cpp:271] Create command done!
[03/23 13:14:23.592576][info][11368][FemtoBoltUvcDevice.cpp:431] init sensor map start!
[03/23 13:14:23.592589][info][11368][FemtoBoltUvcDevice.cpp:458] init sensor map done!
[03/23 13:14:23.593259][info][11368][AbstractDevice.cpp:124] 	- Firmware version: 1.1.0
[03/23 13:14:23.593308][info][11368][FemtoBoltUvcDevice.cpp:275] Init depth process param start!
[03/23 13:14:23.934336][info][11368][MSDEConverterDevice.cpp:781] got nvram data failed.retrying...

[The "got nvram data failed.retrying..." line repeats some 10 thousand times ...]

[03/23 13:14:28.428688][info][11368][MSDEConverterDevice.cpp:781] got nvram data failed.retrying...
[03/23 13:14:28.428746][info][11368][MSDEConverterDevice.cpp:772] got nvram data succeed.
[03/23 13:14:28.428767][info][11368][ObUvcDevice.cpp:112] endpoint:130
[03/23 13:14:28.428828][info][11368][ObUvcDevice.cpp:118] libusb_clear_halt done, endpoint:130
munmap_chunk(): invalid pointer

This issue also happens when allocating an ob::Device in my application that uses the Orbbec SDK.

I am on Linux, with the Orbbec SDK V1.9.5, and firmware version 1.1.0 for the Femto Bolt. Edit: In case it matters, the camera is connected to the PC with the original USB cable.

puzzlepaint avatar Mar 23 '24 12:03 puzzlepaint

I was able to test on another PC as well now using Windows as operating system, and the issue does not seem to occur there. But this remains a large issue on the Linux PC. I have to constantly unplug and reinsert the cameras there to make them work (since the crash does not seem to happen the first time a camera is acquired after being plugged in).

puzzlepaint avatar Mar 25 '24 22:03 puzzlepaint

Could you provide me with the specifications of your computer, such as CPU and GPU models? We'd like to try reproducing the issue.

zhonghong322 avatar Apr 07 '24 02:04 zhonghong322

The specifications of the PC on which I observed the crashes are:

CPU: AMD Ryzen 9 7950X 16-Core Processor
RAM: 64 GB
GPU: NVIDIA GeForce RTX 4090
OS: Ubuntu 22.04.4 LTS (Kernel Version: 6.5.0-26-generic (64-bit))

puzzlepaint avatar Apr 07 '24 16:04 puzzlepaint

I also have this issue. I have experienced it on two separate Windows 11 machines running latest 1.1.2 Femto Bolt firmware. One machine is an Nvidia GPU and the other is an AMD GPU. Both GPUs are well above minimal spec.

For me it is always the second of two Femto Bolts failing to start.

GestureArtAndDesign avatar Jun 25 '24 00:06 GestureArtAndDesign

Hi, we have the same problem using the Orbbec SDK. When allocating a new device this error occurs. Our specs:

CPU: AMD Ryzen Threadripper 1950X 16-Core Processor
GPU: Nvidia GA102 GeForce RTX 3090
RAM: 64 GB
OS: Ubuntu 22.04.4 LTS

EDIT: Powercycling the device helps until the next program restart.

nwrkbiz avatar Jul 01 '24 13:07 nwrkbiz

Hi, same issues here. Randomly, the device-open process gets stuck in an "got nvram data failed. retrying..."-loop indefinetely. Unfortunately with 4 connected devices, it happens like 80% of the time, that one camera gets stuck.

Processor	12th Gen Intel(R) Core(TM) i5-12500   3.00 GHz
Installed RAM	32,0 GB (31,4 GB usable)
System type	64-bit operating system, x64-based processor
Graphics	Intel(R) UHD Graphics 770 (internal)

Edition		Windows 10 IoT Enterprise LTSC
Version		21H2
Installed on	‎31/‎07/‎2023
OS build	19044.5011
Experience	Windows Feature Experience Pack 1000.19060.1000.0

Maybe it's a matter of the IoT-Windows version?

David-Conrad avatar Oct 22 '24 08:10 David-Conrad

I am having the same issue on

Ubuntu 22.04 Linux 6.8.0-47-generic #47~22.04.1-Ubuntu SMP PREEMPT_DYNAMIC Wed Oct 2 16:16:55 UTC 2 x86_64 x86_64 x86_64 GNU/Linux

with AMD Ryzen 9 7950X 16-Core Processor CPU, but not on

Debian 11 Linux 5.10.0-28-amd64 #1 SMP Debian 5.10.209-2 (2024-01-31) x86_64 GNU/Linux

with Intel(R) Core(TM) i9-9900K CPU @ 3.60GHz CPU.

I have noticed that when running the ob_hello_orbbec, the first time, the camera can be opened.

But the second time, I get the [info][123710][MSDEConverterDevice.cpp:786] got nvram data failed.retrying... error.

When I press Ctrl + C while the many got nvram data failed.retrying are printed, then on the next run of the ob_hello_orbbec, the camera can be opened.

So the next time (as the camera could just be opened properly) it will crash again.

It is as if the Ctrl + C resets the USB/library/bus, or "properly" releases the camera.

I am using the pyorbbecsdk, and to make sure I can open the camera, I have implemented the following procedure, which sends Ctrl + C to a subprocess trying to open the camera.

import subprocess
import time

import pyorbbecsdk as ob


def reset_orbbec(sleep=5):
    # This code will crash if the camera is dead
    cmd = "import pyorbbecsdk as ob ; ob.Pipeline()"

    # Trying to open the camera:
    #  - if it succeeds, then, now the camera is dead
    #  - if it fails, then, the camera is already dead
    proc = subprocess.Popen(["python3", "-c", cmd], shell=False, stdout=subprocess.PIPE)
    proc.wait()
    print("Process finished with exit code", proc.poll())

    # We know the camera is dead: let's send Ctrl+C to it
    proc = subprocess.Popen(["python3", "-c", cmd], shell=False, stdout=subprocess.PIPE)
    time.sleep(sleep) # Making sure we wait long enough
    proc.kill()
    print("Process killed")

reset_orbbec()

# The camera can now be opened
ob.Pipeline()

I can then use the Orbbec camera. I sometimes get segmentation errors, also depending on which USB port the camera is plugged.

ocanevet avatar Oct 25 '24 14:10 ocanevet