Bug report: Build hl2ss_ulm.so and main_ulm.cpp

[Secret-Lin]

Thanks for your great work. I am trying to do some image process on what holoLens2 sees. So, I wanted to import hl2ss as shared library(hl2ss_ulm.so). Here are bugs I met and how I solved: For building hl2ss_ulm.so:

1. ‘std’ does not name a type, like mutex; How to fixed: using c++17.
2. ‘wstring_convert’ is not a member of ‘std’; How to fixed: using c++17 and add '#include < l o c a l e >' (delete spaces)in hl2ss_ulm.cpp.
3. Not find defination of hl2ss_mt::source::xxxxx; How to fixed: run 'g++ -fPIC -shared -std=c++17 -o libhl2ss_ulm.so hl2ss_ulm.cpp hl2ss_mt.cpp' to build hl2ss_ulm.so. For building main_ulm:
4. No such file or directory of xxxxx, like opencv or can not find lhl2ss_ulm. Here is the tasks.json of I used: '{ "version": "2.0.0", "tasks": [ { "type": "cppbuild", "label": "C/C++: g++ build active file", "command": "/usr/bin/g++", "args": [ "-fdiagnostics-color=always", "-g", "-std=c++17", "${file}", "hl2ss.cpp", "hl2ss_lnm.cpp", "../../3rdparty/Zdepth/src/.cpp", "../../3rdparty/Zdepth/zstd/src/.c", "-o", "${fileDirname}/${fileBasenameNoExtension}", "-DHL2SS_ENABLE_ZDEPTH", "-I../../3rdparty/Zdepth/include", "-I../../3rdparty/Zdepth/zstd/include", "-I../../3rdparty/Zdepth/zstd/src", "-lavcodec", "-lavutil", "-I/usr/include/opencv4", // OpenCV 4 include path "-L/usr/local/lib", "-L/home/rvc/colcon_ws/utils/hl2ss/extensions/client_cpp", "-lopencv_highgui", "-lopencv_imgcodecs", "-lopencv_imgproc", "-lopencv_core", "-O3", "-lhl2ss_ulm", "-Wl,-rpath,/home/rvc/colcon_ws/utils/hl2ss/extensions/client_cpp", "-lpthread" ], "options": { "cwd": "${fileDirname}" }, "problemMatcher": [ "$gcc" ], "group": { "kind": "build", "isDefault": true }, "detail": "Task generated by Debugger." } ]

}' 2. error: a reinterpret_cast is not a constant expression. In hl2ss_ulm.h, the author uses constexpr to void multidefination, but reinterpret_cast can not be used in the return. How to fixed: here is the code I changed for constexpr functions in hl2ss_ulm.h: 'constexpr map_rm_vlc unpack_rm_vlc(uint8_t* payload) { rm_vlc_metadata* metadata = reinterpret_cast<rm_vlc_metadata*>(payload + parameters_rm_vlc::PIXELS); return { payload, metadata }; // return { payload, (rm_vlc_metadata*)(payload + parameters_rm_vlc::PIXELS) }; } constexpr map_rm_depth_ahat unpack_rm_depth_ahat(uint8_t* payload) { uint16_t* payload_temp = (uint16_t*)(payload); uint16_t* return_value2 = (uint16_t*)(payload + (parameters_rm_depth_ahat::PIXELS * sizeof(uint16_t))); rm_depth_ahat_metadata* metadata = (rm_depth_ahat_metadata*)(payload + (2 * parameters_rm_depth_ahat::PIXELS * sizeof(uint16_t))); return { payload_temp, return_value2, metadata }; }

constexpr map_rm_depth_longthrow unpack_rm_depth_longthrow(uint8_t* payload) { uint16_t* payload_temp = (uint16_t*)(payload); uint16_t* return_value2 = (uint16_t*)(payload + (parameters_rm_depth_longthrow::PIXELS * sizeof(uint16_t))); rm_depth_longthrow_metadata* metadata = (rm_depth_longthrow_metadata*)(payload + (2 * parameters_rm_depth_longthrow::PIXELS * sizeof(uint16_t))); return { payload_temp, return_value2, metadata }; }

constexpr map_rm_imu unpack_rm_imu(uint8_t* payload) { rm_imu_sample* payload_temp = (rm_imu_sample*)(payload); return { payload_temp }; }

constexpr map_pv unpack_pv(uint8_t* payload, uint32_t size) { pv_metadata* metadata = (pv_metadata*)(payload + size - sizeof(pv_metadata)); return { payload, metadata }; }

constexpr map_microphone_raw unpack_microphone_raw(uint8_t* payload) { int16_t* payload_temp = (int16_t*)(payload); return { payload_temp }; }

constexpr map_microphone_aac unpack_microphone_aac(uint8_t* payload) { float* payload_temp = (float*)(payload); return { payload_temp }; }

constexpr map_microphone_array unpack_microphone_array(uint8_t* payload) { float* payload_temp = (float*)(payload); return { payload_temp }; }

constexpr map_si unpack_si(uint8_t* payload) { si_frame* payload_temp = (si_frame*)(payload); return { payload_temp }; }

constexpr map_eet unpack_eet(uint8_t* payload) { eet_frame* payload_temp = (eet_frame*)(payload); return { payload_temp }; }

constexpr map_extended_audio_raw unpack_extended_audio_raw(uint8_t* payload) { int16_t* payload_temp = (int16_t*)(payload); return { payload_temp }; }

constexpr map_extended_audio_aac unpack_extended_audio_aac(uint8_t* payload) { float* payload_temp = (float*)(payload); return { payload_temp }; }' Finally, I successfully built main_ulm and saw the pv. Thanks again for the author's great work. And these bugs may due to my ubuntu20.04 foxy env. Hope this issue can help someone.

[jdibenes]

Very cool. Thank you. Indeed, I only tested the client shared library on Windows and MSVC.

[Secret-Lin]

@jdibenes I figure that you do not write a corresponding hl2ss_3dcv for cpp, I want to transfer the value of longthrow to depth refer to its camera in cpp. By reading your sample_pv_depth_lt.py, I have following questions:

1. Are the world coordinates of pv and lt the same one?
2. Why you do transformation on xy1_o and xy1_d seperately and finally hstack them togather?

(By the way, I cannot run sample_pv_depth_lt.py, it just stuck and cannot be killed. And I can run client_steam_*.py to see the sensors' information)

[jdibenes]

1. It is the same coordinate frame but have different transforms wrt rig node.
2. To extract depth from xy1_o and for readability. Some of the transforms could be combined for efficiency.
3. Try import hl2ss_imshow before import hl2ss.

[Secret-Lin]

Thanks for replying, this work well:

3. Try import hl2ss_imshow before import hl2ss.

When I test result of sample_pv_depth_lt.py, I found that you use hl2ss_3dcv.pv_fix_calibration(pv_intrinsics, pv_extrinsics) to generate color_extrinsics where pv_extrinsics is np.eyes. And color_extrinsics is [[ 1. 0. 0. 0.] [ 0. -1. 0. 0.] [ 0. 0. -1. 0.] [ 0. 0. 0. 1.]] My question is why not use extrinsics from calibration_pv (calibration_pv = hl2ss_3dcv.get_calibration_pv(host, hl2ss.StreamPort.PERSONAL_VIDEO, calibration_path, 1000, pv_width, pv_height, pv_fps) from #134 )? Here is the calibration_pv.extrinsics: [[ 0.0281502 -0.9993722 0.02151053 0. ] [-0.99930006 -0.0276047 0.02524317 0. ] [-0.02463353 -0.02220608 -0.99944973 0. ] [-0.0494109 0.00193774 0.01321972 1. ]] I test the result using calibration_pv.extrinsics to replace color_extrinsics, the max diff of pv_depth and pv_depth_origin is 0.018292844 for a fixed holoLens facing a black flat wall with around 0.3m.Maybe the diff is acceptable.

I test the final pv_z and pv_z_origin, the max diff is 0.44158256 which is not acceptable. And need change v0:v1 to v1:v0 as v1<v0.