Expose the playback and record APIs through C#

[MsDerekMa]

Describe the bug The Playback and Record APIs are not currently exposed in the C# API.

Expected behavior The Playback and Record APIs should be wrapped so that they are accessible through the C#APIs. In addition they should have appropriate tests to ensure that the functionality works.

Sub tasks

• [ ] Playback API parity
• [ ] Record API parity
• [ ] Unit tests
• [ ] Documented public API surface

[trekze]

Is this likely to be worked on soon? I'm starting work on a project and I'd very much prefer to do it in C#, but if there's a chance this will be pushed back for more important features, I might take the hit and resort to C, which I haven't used in a while.

Thanks!

[ahoink]

is this still on track to be included in the 1.4.0 release?

[natelowry]

bumping this again and lmk if there's anything i can do to help 😸

this will be awesome when it's added. ❤️

[Thebinoman]

This would be EXTREMELY helpful. Please add this ASAP 🙏

[natelowry]

@Thebinoman agreed it'll be great, but in the meantime, you can get really far with a light wrapper around the native methods (see https://github.com/microsoft/Azure-Kinect-Sensor-SDK/blob/develop/src/csharp/SDK/Native/NativeMethods.cs) and some reflection to get the native handles. Enough so that we've shipped with that code and not really had any recording issues.

Happy to post some example code if that's helpful.

[Thebinoman]

@Thebinoman agreed it'll be great, but in the meantime, you can get really far with a light wrapper around the native methods (see https://github.com/microsoft/Azure-Kinect-Sensor-SDK/blob/develop/src/csharp/SDK/Native/NativeMethods.cs) and some reflection to get the native handles. Enough so that we've shipped with that code and not really had any recording issues.

Happy to post some example code if that's helpful.

Thank you. I wasn't aware of that. I saw the namespace Microsoft.Azure.Kinect.Sensor.Native, but couldn't figure out how to use it. I haven't found documentation about that. Good to know it already been implemented.

Yes, code examples would be great, especially of how to get body tracking to work with playback.

BTW, I'm using unity, are you aware of any issues or gotchas on this (body tracking with playback from an .mkv file)?

Thanks!

[natelowry]

all of this was based on thedewi's amazing work in #1711 (which is based on Brent-A's work in #822)

here's what i think the minimum steps are for you to get a recording going:

1. add NativeMethods.cs to your project
2. add the k4a_record_create, k4a_record_write_capture, and k4a_record_flush DLLImports to NativeMethods (or use this version of the file that already has all the recording methods)

[DllImport("k4arecord", CallingConvention = k4aCallingConvention, CharSet = CharSet.Ansi)]
public static extern k4a_result_t k4a_record_create(string path, IntPtr device, k4a_device_configuration_t deviceConfiguration, out k4a_record_t handle);

[DllImport("k4arecord", CallingConvention = k4aCallingConvention)]
public static extern k4a_result_t k4a_record_write_capture(k4a_record_t handle, IntPtr capture);

[DllImport("k4arecord", CallingConvention = k4aCallingConvention)]
public static extern k4a_result_t k4a_record_flush(k4a_record_t handle);

3. to start recording, create a native device config for recording format

var deviceConfig = new NativeMethods.k4a_device_configuration_t
{ //example values
    camera_fps = FPS.FPS30,
    color_format = KinectImageFormat.ColorMJPG, 
    color_resolution = ColorResolution.R1536p,
    depth_mode = DepthMode.NFOV_Unbinned,
    synchronized_images_only = true
};

4. assuming you have a Device called _kinect, in an unsafe block, get the device handle/pointer and pass that into k4a_record_create and write the recording header (save the _recordingHandle for writing frames)

unsafe
{
    var deviceHandle = typeof(Device).GetField("handle", BindingFlags.Instance | BindingFlags.NonPublic).GetValue(_kinect);
    var deviceTypeHandle = Type.GetTypeHandle(deviceHandle);

    var deviceDangerousHandle = Type.GetTypeFromHandle(deviceTypeHandle).GetMethod("DangerousGetHandle", BindingFlags.Instance | BindingFlags.Public).Invoke(deviceHandle, null);

    var result = NativeMethods.k4a_record_create(recordingPath, (IntPtr)deviceDangerousHandle, deviceConfig, out _recordingHandle);
    
    result = NativeMethods.k4a_record_write_header(_recordingHandle);
}

5. as you receive captures (frames), do the same for the current capture (assuming you have a Capture called _capture)

unsafe
{
    var captureHandle = typeof(Capture).GetField("handle", BindingFlags.Instance | BindingFlags.NonPublic).GetValue(_capture);
    var captureTypeHandle = Type.GetTypeHandle(captureHandle);
    var captureDangerousHandle = Type.GetTypeFromHandle(captureTypeHandle).GetMethod("DangerousGetHandle", BindingFlags.Instance | BindingFlags.Public).Invoke(captureHandle, null);
    var result = NativeMethods.k4a_record_write_capture(_recordingHandle, (IntPtr)captureDangerousHandle);

    //not sure if we should be flushing every frame
    result = NativeMethods.k4a_record_flush(_recordingHandle);
    captureHandle = null;
    captureDangerousHandle = null;
}

6. stop recording with k4a_record_close

var result = NativeMethods.k4a_record_flush(_recordingHandle);
NativeMethods.k4a_record_close(_recordingHandle.DangerousGetHandle());
_recordingHandle = null;

can't speak to Unity, but don't see why it wouldn't work. if you find gotchas be sure to share them with the class :)

[Thebinoman]

@natelowry Thank you, and sorry for the very late response, I'll try that. Looks promising.

However, do you know how to play the recorded file?

[natelowry]

same deal using k4a_playback_open and k4a_playback_get_next_capture.

see the rest of the things here: https://microsoft.github.io/Azure-Kinect-Sensor-SDK/master/structk4a__playback__t.html

dig into those PR's referenced above as they have all the code you need.

[thedewi]

Thanks @natelowry.

Very late reply, but you should be able to build my PR #1711 directly, and call Playback.Open(), playback.GetNextCapture(), etc (without adding any more PInvoke declarations). It is binary compatible with the native libraries in the SDK tools folder. But it's still at SDK version 1.1.0; I haven't updated it for the latest yet.