Environment-Aware WebXR Augmented Reality

• Web-based framework for environment-aware rendering and interaction in augmented reality based on WebXR using three.js
• The framework implements realistic rendering of 3D objects, handles with geometry occlusions, matches the lighting of the environment, casts shadows, and provides physics interaction with real-world objects.
• Target the web environment and have designed our solution to work on a vast range of devices as, monocular camera setups or devices with dedicated higher quality depth sensors.
• Single application while being capable of obtaining over 20 frames per second even on middle-range devices.
• Demonstration video of the features in the framework.
• Based on the experimental WebXR AR draft API available in Chrome 88+
  • Lighting Estimation
  • Depth Sensing Module
  • Hit Test Module

Getting Started

• Download the repository from git gh repo clone tentone/ar-occlusion
• Install Node and NPM.
• Prepare a browser compatible with WebXR in a depth with support for depth-estimation, hit-test and lighting.
• Install dependencies from NPM by running npm install and start the code running npm run start

Algorithms

• To ensure consistency in the rendering of the virtual scene we implement a physically-based rendering pipeline.

• Physically correct attributes are associated with each 3D object that, combined with lighting information captured by the device, enables the rendering of AR content that matches the real-world illumination.

• Depth data provided by WebXR to dynamically place 3D objects around the environment, calculate geometry occlusion and build a model of the environment to enable physics interaction.

• Calculate geometry occlusion on a pixel basis directly from the projected vertices coordinates to prevent loss in data precision

• To reuse the PBR pipeline of three.js we inject our code into the existing shaders generated by the library using the onBeforeCompile method of the material.
• We use the cannon.js physics simulation engine for all physics interactions.
• The environment is mapped using a probabilistic voxel based model that is updated every frame.

License

• The code from the project is MIT licensed. The license is available on the project repository,