Camera Intrinsics and Spatial Coordinates Update

[camnewnham]

Hi all, Thanks for the fantastic plugin - it works exceptionally well. I've added a few things that you might be interested in integrating -

1. Setting the video type to 'VideoRecord'
2. Allowing the stream to be shared
3. Adding camera intrinsics
4. Accepting SpatialCoordinateSystem as well as IntPtr to work around Unity bug.

More detailed description and reasoning of the above:

VideoRecord instead of VideoPreview I noticed in the comments there was some uncertainty about the difference between the two. After quite a bit of testing I found the following:

• VideoRecord does not crush the display quality in the same way VideoPreview does when in use.
• VideoRecord does not show the recording indicator (despite attempts to have that as an option) This (as far as I can tell) matches the Vuforia implementation as the effects seem similar.

Shared Stream This allows use one to record while the camera is active. Again, appears to match the Vuforia implementation - it works with device portal recording, on device recording or Miracast streaming in RS5, but the live stream via device portal still conflicts.

Camera Intrinsics This allows retrieval of the distortion coefficients for use in image processing scenarios (i.e. OpenCV) Initial tests with Windows.Media.Devices.Core.CameraIntrinsics worked well - but the API is in prerelease and appears to cause problems with distribution. After a bit of digging, the properties can be retrieved via GUID, as with the spatial coordinate system etc.

Accepting SpatialCoordinateSystem instead of IntPtr Due to Unity bug 1025171 NET Native builds don't run when setting the world origin pointer (please vote on this issue!). A workaround for this is to accept the SpatialCoordinateSystem directly - which you can cache when you load a scene and Unity refreshes it anyway. Implementation details in code comments.

As a side note, 1344x756 works excellently for image tracking as you have some peripheral pixels - wider than the HoloLens FOV so you don't lose your objects as soon as they're slightly out of frame.

Happy to discuss changes or future improvements.

[motionsmith]

Hey @camnewnham. What version(s) is this compatible with? I think master works in V 5.6, unless you've made it compatible with newer version. I no longer support this project, but lots of people still try to use it. I want to approve this pull request, but I'd like to also update the documentation...this really needs to get working in the newer versions of Unity

[motionsmith]

Also, these changes look really useful. It looks like you really know what you're doing. Thank you.

[camnewnham]

@motionsmith these changes were tested only on Unity 2017.1.3p1 - I would assume compatibility up to at least 2017.4.x but this has not been tested.

@EnoxSoftware might also also be interested in updating the OpenCVForUnity examples as these changes allow getting the camera intrinsics on the fly rather than recomputing.

When I eventually get around to updating some of my projects to 2018.x I'll be happy to update this for compatibility if doesn't work out of the box - just waiting (as usual) for things to be slightly more stable between Unity and UWP/HoloLens.

[motionsmith]

[Deep breath] Thanks @camnewnham! I don't have time to test these right now, but these look like such good changes. Going to merge.

[RawadBahsas]

hey @camnewnham, did you test these changes on a device that has already been updated. My hololens is currently running on this OS https://www.catalog.update.microsoft.com/ScopedViewInline.aspx?updateid=8ab30b7d-e14d-426c-8cbf-c0e8114b46a8 and I can't make your branch work. it gives me a white box and the feed is never there then the application crashes in like 5 seconds. and i am working on Unity 2017.1.3p1. any ideas?

[camnewnham]

Hi @RawadBahsas, I hadn't actually tested the examples because I was working straight into a separate app.

I've updated the VideoCapturePanel example - the main issue seemed to be stemming from InvokeOnAppThread - so I've replaced it with a simple Queue.

Note that there's no handling in the example for permissions being denied, so you might need to launch it once, accept permissions, and then if it crashes launch it again with permissions already accepted.

[RawadBahsas]

Hi @camnewnham, Thanks for the update, I was able to make the projection example work as well. I use the projection example to build a facial recognition experience on Hololens. my application shows a rectangle around a persons face then it sends the face bytes to a server and the server replies with a name and a business card photo of that person witch I then show above his/her face.

Currently my application is working on unity 5.6.3p2

if you have time, I would like you to explain to me how your updates fixed the spatial mapping issues that appeared after June's update. (my application couldn't get the spacial mapping correct if the hololens didn't have the whole room scanned before launching the app, and the matrices went funny when looking towards unscanned areas )

anyway its working fine after your updates. cheers

[camnewnham]

The issue that they were addressing was only relevant to Master builds, where the code required to use the native pointer for the SpatialCoordinateSystem was being stripped by the editor. This doesn't affect debug and release builds, so if your issue still occurs there it's unrelated.

I haven't experienced any issues with incorrect coordinates - you do need to set the spatial coordinate pointer in this library to get the correct camera matrices though.

App sounds interesting - is it using OpenCVForUnity?

[RawadBahsas]

Am only using windows SDK to get the faces and return it with every frame using VedioCaptureSample. then in the app I created a basic tracking algorithm to keep tracking of a person and collect samples of his face and try to recognize them by calling a webApi to my face recognition server. OpenCVForUnity isn't free after all

Am trying to make the the stream shared by setting _SharedStream to true with no luck. I just wanna be able to open the hololens portal live preview while using the app. any idea why am not able to do so? BTW I tried it on this version not mine, so i basically just changed the SharedStream to true nothing else

[camnewnham]

I think the shared streaming only works on RS5 - this issue also affects Vuforia in RS4 but didn't in RS1. (Though Miracast will be a healthy alternative in RS5 anyway).

I think it works best when device portal stream is started first, and then you match the resolution (I.e. 1280x720 at 30fps when on the hq setting) - you won't be able to share it if you're requesting different formats as far as I know.

[christjt]

Hello, great work and seems like a good addition! I was wondering if you have some insight for the camera intrinsic with preview vs. VideoRecord. In addition to the differences that you listed the preview and video record streams are the same frames but from different steps in the graphics pipeline of the HoloLens. The frames that come from the video record have been undistorted (pinhole camera causes distortion). This begs the question if one should trust the distortion data from the intrinsic matrix. These distortion values are used by the graphics pipeline to undistort the frames for the video record, while the preview frames remain "raw" and distorted.

This means that when doing image processing one should assume that all distortion coefficients to be 0 with frames from the video record, as the images have been undistorted (given perfect camera calibration, which it definitely isn't). My point is that using these distortion values for the video record stream may be incorrect. Also, when doing undistortion it is not uncommon to center the focal point. I am not sure exactly about the all the steps the graphic pipeline of the HoloLens does (don't think this is disclosed anywhere), but one should be aware of these things if you are trying to get the best precision possible. The enox software that you are referring to used VideoPreview instead of VideoRecord, probably because of this.

The safest, but also most cumbersome is doing your own camera calibration and not relying on the data from the API. Hopefully they do a better job with camera calibration on the HoloLens 2. I can also confirm that they indeed do calibration per HoloLens as the intrinsics from the API differ from HoloLens to HoloLens.

[camnewnham]

Thanks Chris, and some interesting points. I haven't done any testing without the device provided projection and distortion matrices, so I can't say for sure whether there would be an improved result in assuming an undistorted frame. In my usage using VideoRecord - which is typically spatially locating QR codes and Aruco markers, the provided values appear to work correctly (to within a few mm). It's also a little hard to say because the processing overhead on the HoloLens and using a slightly inefficient OpenCV implementation also mean a ~200ms lag - particularly when requesting the wider FoV frames available from the camera which are much more practical. I am certainly no expert in camera/graphics processing - much more of a tinkerer just trying to get things to work! We also run this on a wide range of HoloLens where doing per-device calibration isn't a viable option.

[techtide]

Regarding the questions of compatibility, does this PR or the plugin itself (master) work on the 2018/2019 builds of Unity? @motionsmith @camnewnham

[camnewnham]

Hi @techtide, this PR works in 2018. I can't say if the example still runs, but the .dll is definitely functional. Since it has no reference to Unity, it should only break if UWP libraries change, which seems very unlikely.

Though I would note that the camera APIs for HoloLens 2 work a little differently as they require camera profiles - https://docs.microsoft.com/en-us/windows/mixed-reality/locatable-camera

Cheers

[cookieofcode]

This PR works fine on branch upgrade_2018_2 in Unity 2019.4.11f1 to capture video frames using the HoloLens 2. The examples are missing some scene configurations after the Unity upgrade.