Modifier keys not working with visionOS virtual desktop

[YANOUSHek]

Hi. I'm having issues with using my Vison Pro remote desktop and then trying to control the visionOS apps with the keyboard and mouse connected to my mac. The problem is that all the modifier keys (shift, command etc.) are not working when trying to type anywhere in a visionOS app. This goes away when I enable the "Full keyboard access" in accessibility settings in visionOS, but that comes with own issues.

I've noticed today that when I quit Karabiner Elements running on my mac the keyboard start working fine. When I start Karabiner again the modifier keys stop working. Would anyone consider working on this it makes the virtual desktop almost useless when I can't have my keyboard mappings.