[MRTK3] Design Polish - ObjectBar, See-it, Say-it label, Hand Interaction Examples scene

[cre8ivepark]

Overview

• ObjectBar prefabs margin updated to align with recent button prefab updates.

• See-it, Say-it label for both canvas and non-canvas buttons. Showing & hiding with StateVisualizer.

• Hand Interaction Example scene updates -- Magic window example for touch interaction example -- Added WhiteboardExample prefab with description (indirect interaction with custom interactions) -- Added Canvas UI example (dashboard) with description

• Color option buttons added to WhiteboardExample prefab.

• Non-canvas Toggle Button prefab color fix. Updated to use StateVisualizer animation

Assets info

• WhiteboardMaterial.mat updated to rounded corners & proximity light
• MagicWindowBackplate_190x62.mat: Material for magic window, configured with Stencil Testing, Stencil Comparison 'GreaterEqual', Stencil Operation 'Replace'
• MagicWindow_190x62.fbx: Magic window geometry with rounded corners
• MRTK_Standard_White_StencilObject.mat: Material for object inside the magic window. Configured with Stencil 'LessEqual' and 'Keep' to make it only visible through the magic window. Used for MRTK 3D logo and number 3 objects
• MRTK_Standard_LogoGradient: Goggle area of the MRTK 3D logo with stencil effect
• MRTK_SeeItSayItLabelPlate_13mm_Gradient.mat: Backplate material optimized for small label size
• CanvasBackplate_SeeItSayIt.mat: Backplate material optimized for small label size (used with CanvasElementRoundedRect)
• MRTK_Stageplate_1cm.mat: Backplate material for thick stage/table shape. Used in HandInteractionExamples scene
• MRTK_Logo_Number3.glb: Number 3 shape model

Screenshots

ObjectBar prefabs margin updated to align with recent button prefab updates.

See-it, Say-it label fixed for non-canvas button prefabs. Material fix, show & hide with StateVisualizer

See-it, Say-it label added to canvas button prefab (Action Button.prefab)

• Label show & hide animation added to NonCanvasButtonActiveFocus.anim
• Before & After:

https://user-images.githubusercontent.com/13754172/179468556-7523cc0e-4ace-41b4-b726-c498a476f9be.mp4

• Smooth show & hide with the alpha transition will be added later when below MRGT PR updates are available for MRTK3
• https://github.com/microsoft/MixedReality-GraphicsTools-Unity/pull/76

Hand Interaction Example scene updates

• Added WhiteboardExample prefab with description (indirect interaction with custom interactions)
• Color option buttons added to WhiteboardExample prefab. Added proximity light to the Pen.prefab
• Added Canvas UI example (dashboard) with description

• Replaced touch examples with Magic window example. MagicWindowExample.prefab (MRTK 3D logo rotates on touch)

Toggle button state color fix (non-canvas)

• Created NonCanvasToggleSwitchBackplate.anim which controls the Fade Out property
• Updated NonCanvasAuthoringController.controller for in-editor validation & testing

[Zee2]

@cre8ivepark , does the see-it-say-it design have signoff from Ramiro? It looks a little, odd, in my opinion... it also doesn't fit any of the variants in the Figma.

[Zee2]

@cre8ivepark , is there some good way we can make the see-it-say-its optional/opt-in, say, perhaps only when a speech interactor/system is available?

[cre8ivepark]

@cre8ivepark , does the see-it-say-it design have signoff from Ramiro? It looks a little, odd, in my opinion... it also doesn't fit any of the variants in the Figma.

Synced with Alex and decided to go with this design since it is a natural alignment & transition from the version we had in Sydney + rounded pill style update gives good visual separation from other UI backplate elements, still aligned with rounded corner visuals. And we need the backplate for the text legibility.

[cre8ivepark]

@cre8ivepark , is there some good way we can make the see-it-say-its optional/opt-in, say, perhaps only when a speech interactor/system is available?

That would be a great improvement. Is it something you can help with? Directly in this PR or as a separate PR? It seems it requires some technical tweaks since it involves a Speech interactor + StateViz control.

[Zee2]

We shouldn't add a ProximityLight to the pen, because it will interfere with the maximum-of-two proximity light count.

[Zee2]

I don't think we should add the big Canvas example to the hand interaction example scene. It could cause performance issues, especially if we're trying to compare the v2 hand interaction example to v3. (v2 didn't have a large multi-layered UI panel in the scene, so it's a bit of an unfair comparison)

[Zee2]

Following our previous discussion, I do think that the SeeitSayit should start out as a separate component for now, and let people add them if they want. Then, we can add the seeitsayit to the actual button prefab once we figure out a good way to only show them on demand. (This will reduce the breakiness of the prefab changes.)

[cre8ivepark]

• Removed See-it, Say-it label property from both canvas & non-canvas buttons.
• Removed the Proximity light from the Pen prefab.

[cre8ivepark]

I don't think we should add the big Canvas example to the hand interaction example scene. It could cause performance issues, especially if we're trying to compare the v2 hand interaction example to v3. (v2 didn't have a large multi-layered UI panel in the scene, so it's a bit of an unfair comparison)

We did have UGUI layout examples and volumetric canvas button examples in the scene. I see your perf concerns. Do you think the new one is much more complex compared with the previous ones? Since this scene is the most widely used MRTK example scene for investigations and demos, it would be great to showcase canvas-based volumetric UI here which is much better than previous examples in MRTK2 :)
Actual perf data will be helpful for this. I'll try on-deivce profiler and see if we need to simplify the canvas example to keep the scene running on 60fps. Thanks!

[RogPodge]

I don't think we should add the big Canvas example to the hand interaction example scene. It could cause performance issues, especially if we're trying to compare the v2 hand interaction example to v3. (v2 didn't have a large multi-layered UI panel in the scene, so it's a bit of an unfair comparison)

We did have UGUI layout examples and volumetric canvas button examples in the scene. I see your perf concerns. Do you think the new one is much more complex compared with the previous ones? Since this scene is the most widely used MRTK example scene for investigations and demos, it would be great to showcase canvas-based volumetric UI here which is much better than previous examples in MRTK2 :) Actual perf data will be helpful for this. I'll try on-deivce profiler and see if we need to simplify the canvas example to keep the scene running on 60fps. Thanks!

I think perf wise it should be fine, especially with the relatively small size of the UI example.

MRTK2 perf snapshot:

MRTK3 perf snapshot:

[RogPodge]

I noticed that the whiteboard allows for every interactor to interact with it, which leads to some pretty weird drawings when inadvertently touching the board due to the poke interactor glancing against it, or the gaze interactor accidentally triggering a gaze-pinch.

Looking to alleviate this by creating a new "Whiteboard" interaction layer and changing our interactors to only target the Default Interaction Layer Mask

[Zee2]

@cre8ivepark , it looks like the Whiteboard UI changes aren't built with Canvas... do we want to take this time to redo that Whiteboard UI in Canvas?

[RogPodge]

Resolved offline. Reverting rig prefab changes back to "Everything" and then apply a scene override to prevent Whiteboard interactions.

Implemented the changes here. Everything with the whiteboard should be as it was prior to: https://github.com/microsoft/MixedRealityToolkit-Unity/pull/10751/commits/8a3b1f2f0240e6bd5386e7231622bbe74ca2638e.

Some other changes were made, the end result is as follows:

• The pen should be able to interact with all objects, acting like a poke interactor on the end of the pen
• The whiteboard now does not accept interaction from IGaze, IRay, and the CanvasProxyInteractor. This is because
• Gaze interaction does not give feedback that the user is currently targetting the whiteboard, and does not give good feedback as to where the user is initiating their draw
• Far Ray Interaction locks the cursor to the whiteboard while the drawing occurs, making the motion of drawing unintuitive
• Interacting using just the mouse cursor via the CanvasProxyInteractor has inconsistent and unintuitive behavior once the mouse cursor goes off the canvas while holding down a click.

[cre8ivepark]

@cre8ivepark , it looks like the Whiteboard UI changes aren't built with Canvas... do we want to take this time to redo that Whiteboard UI in Canvas?

Updated Whiteboard to Canvas UI. Question: is it possible to make the whiteboard Quad expandable and fill a canvas area? I couldn't figure out how to do it so manually sized for now.

[RogPodge]

I spent some time trying to figure out if it was possible to use an Image or RawImage instead of a quad to facilitate the drawing code, but couldn't figure out how to do it. I think for the sake of the example it's ok to leave the scaling kind of wonky, though if we were to bring the canvas into MRTK as a new supported UX component, we need to do more work to figure out how to get it scaling nicely within a canvas layout.

[cre8ivepark]

Our non-UI examples didn't provide visual feedback for hover/focus/select. Added visual feedback example to Earth and Heart objects using StateVisualizer for better Gaze + Pinch and other interactions. The outline effect is using MRGT's MeshOutline script and thickness animation is controlled by StateViz. Brought some outline materials from MRTK2. Selectedness amount applied to Rim Light property's power to highlight the object on pinch.

https://user-images.githubusercontent.com/13754172/183239545-3276ffd1-ed4a-4046-9447-dd66b2fd0184.mp4

https://user-images.githubusercontent.com/13754172/183251703-4e4c2fb2-6d73-4982-b3aa-572b1d06bdd1.mp4

[cre8ivepark]

Oh no :( The piano no longer has any sound!

Oh, I found that only mouse input was not working because the audio was assigned to the individual select events (poke, ray, gaze pinch). Removed them and reassigned the event to XRI Events > Select Entered. Now mouse click also works well. Piano prefab updated.

Before

After