Sending a text message in a room freezes people in VR

[matthewbcool]

Description When anyone in room sends a text message via chat, it freezes the screen and locks those in VR.

To Reproduce Steps to reproduce the behavior:

1. create a room and invite people to join in VR
2. send a text
3. Ask VR folks how their experience is.

Expected behavior texts do not lock.

Hardware

• Device: Quest 2

Additional context A work around you can use is 'spawning' the text:

https://user-images.githubusercontent.com/4493657/235373813-aa313187-b3a7-4bf2-82c0-04120d8471d7.mp4

[matthewbcool]

This remains a huge issue for folks in VR. Is this fixed with the new loader? @msub2 encountered it on an older client in Hubs Cloud the other day.

[msub2]

Specifically using the Hubs Cloud 1.17 template, though as Matt mentions this has been an issue for quite a while now. And to clarify the effect even more, it completely halts rendering for anyone in VR for upwards of 5-10 seconds sometimes (if you've ever had an immersive WebXR experience lock up so that you could look around and see black around the last projected image, that's what's happening).

[johnshaughnessy]

There are no commits touching src/react-components/chat-message.js for a long time before April:

So I don't think this is due to any recent change we made directly to our code. Given that @msub2 has observed that this bug has existed for some time, perhaps we introduced the regression back when we updated a bunch of dependencies in order to bump webpack. Specifically, it looks like we bumped html2canvas at ec265eaf33113abd3f070e7fb6dcdc7dccc1fea6 :

-   "html2canvas": {
-     "version": "1.0.0-rc.7",
-     "resolved": "https://registry.npmjs.org/html2canvas/-/html2canvas-1.0.0-rc.7.tgz",
-     "integrity": "sha512-yvPNZGejB2KOyKleZspjK/NruXVQuowu8NnV2HYG7gW7ytzl+umffbtUI62v2dCHQLDdsK6HIDtyJZ0W3neerA==",
-     "requires": {
-       "css-line-break": "1.1.1"
-     }
-   },
+   "html2canvas": {
+     "version": "1.4.1",
+     "resolved": "https://registry.npmjs.org/html2canvas/-/html2canvas-1.4.1.tgz",
+     "integrity": "sha512-fPU6BHNpsyIhr8yyMpTLLxAbkaK8ArIBcmZIRiBLiDhjeqvXolaEmDGmELFuX9I4xDcaKKcJl+TKZLqruBbmWA==",
+     "requires": {
+       "css-line-break": "^2.1.0",
+       "text-segmentation": "^1.0.3"
+     }
+   },

Raw links:

• https://raw.githubusercontent.com/mozilla/hubs/f856e60dd9bab4a0aff063028390a3021156d1f3/package-lock.json
• https://raw.githubusercontent.com/mozilla/hubs/ec265eaf33113abd3f070e7fb6dcdc7dccc1fea6/package-lock.json

I'm not set up to do VR testing, but it may be worth trying out older or newer versions of html2canvas to see if the issue gets fixed. Or, it may be a red herring.

[johnshaughnessy]

I misunderstood the bug report. I watched the video and thought that the issue was related to spawning a rendered text image.

The bug seems to be about handling regular text messages. Please disregard my previous comment.

[msub2]

I did a quick profiling session with my headset and chat-message.js does actually appear to be where the blocking is happening. I have a suspicion on what might be happening just from a quick glance, but I'll set up a local dev environment later today and let you know for sure.

[msub2]

So yes, it does look like html2canvas is completely blocking while it's rendering the text bubble in createInWorldLogMessage. Bumping to the latest version shows no changes. Confusingly, spawning in a chat message as an asset through spawnChatMessage doesn't show the same lag behavior despite both of them calling renderChatMessage.

[takahirox]

Thanks for the investigation.

Random thoughts: I haven't taken a look at the implementation of html2canvas at all, but if the blocking is caused by rendering to canvas, hacking html2canvas to use OffscreenCanvas for offloading from main thread might solve the problem?

Confusingly, spawning in a chat message as an asset through spawnChatMessage doesn't show the same lag behavior despite both of them calling renderChatMessage.

That sounds weird... Would you mind checking (with breakpoint in debugger or anything else) whether the parameters passed to renderChatMessage or html2canvas from createInWorldLodMessage are valid?

[msub2]

Apologies for the delay, I think I understand the source of my confusion though: in the case of spawnChatMessage, the object is being rendered and then added to the scene by the user sending the chat message, not the user in VR, so they don't experience the lag in that case.

I think your idea to use OffscreenCanvas is also a good one, seems like there's been talk on the repo before about it as well: https://github.com/niklasvh/html2canvas/issues/2643 https://github.com/niklasvh/html2canvas/issues/2880