Is it possible to get just one frame point cloud buffer？

[Bornblack]

Hi! petergu I want to send just one frame point cloud buffer to my pc through TCP for point cloud registration. Is it possible? I fail to get the point cloud buffer after an air-tap action. It returns nothing.

My code：

...
#if ENABLE_WINMD_SUPPORT && !UNITY_EDITOR
using HL2UnityPlugin;
#endif

...
#if ENABLE_WINMD_SUPPORT && !UNITY_EDITOR
HL2ResearchMode researchMode;
#endif

...
private float[] PointCloudFloatArray = null;
private byte[] PointCloudBuffer = null;

...
#if ENABLE_WINMD_SUPPORT && !UNITY_EDITOR
public byte[] GetDepthSensorData()
{
    while (researchMode.PointCloudUpdated())
    {
        PointCloudFloatArray = researchMode.GetPointCloudBuffer();
        if (PointCloudFloatArray != null && PointCloudFloatArray.Length > 0)
        {
            Debug.Log("Get access to point cloud buffer" + PointCloudFloatArray.Length);
            // float[] ->byte[] for TCP
            PointCloudBuffer = new byte[PointCloudFloatArray.Length * sizeof(float)];
            Buffer.BlockCopy(PointCloudFloatArray, 0, PointCloudBuffer, 0, PointCloudBuffer.Length);
            researchMode.StopAllSensorDevice();
            break;
        }
        else
        {
            Debug.Log("Cannot get point cloud buffer");
        }
    }
    return PointCloudBuffer;
}
#endif

The InitializeDepthSensor() and StartDepthSensorLoop() are done in Start() function. The restricted capability is added to the manifest file.

Look forward to your reply.

[petergu684]

It would be easier if you can put the error messages here or describe the problem in more details. By "it returns nothing", does it mean that it returns an empty float array so that it shows "Cannot get point cloud buffer" message? If so, you might want to check whether the sensors are initialized successfully in the Start function (any exception raised?). There are a few steps to help you identify the problem. I assume you can already run those official samples without any problem. You can start with some easy functions, such as PrintResolution, PrintDepthExtrinsics and see whether these functions could return valid results. Also, those debug output functions should tell you something as well.

[agroenenberg]

First of all, thanks @petergu684 for sharing this git, you're really helping me out with my thesis by posting this. I wrote a normal DLL but it won't include the winrt components. I'm looking for something exactly like this. I'd like to store one frame of a point cloud (preferably as a .tar file just like the RM sample apps) for processing and registration on my laptop. Do you think that could be possible with your code?

[petergu684]

@agroenenberg Yes, I could visualize the point cloud at >40 fps with the code I posted here so saving one frame is definitely possible. I am not familiar with saving data as tar file. You could probably implement it yourself or referring to their sample code. You just need to make some modifications in HL2ResearchMode.idl, HL2ResearchMode.h, HL2ResearchMode.cpp. If you don't want to make changes in C++, you can also send the point cloud to your laptop via TCP socket and save the data using Python on your laptop. Btw, I followed the Unity's left hand coordinate system in the code here because I want to visualize it in Unity. If you need to keep the original RHS coordinate system, just delete the negative sign here.

[agroenenberg]

@petergu684 Thanks I changed it in C++ and now it works! I still have to figure out how to save only one frame, but thanks for the recommendations!