Immersive Gaussian Splatting

Immersive Gaussian Splatting explores a mixed-reality workflow for environment capture, 3D reconstruction, spatial inspection, and asset export. A Meta Quest 3 application captures calibrated RGB frames with camera intrinsics and headset poses, guides the user through setup, capture, and inspection, and renders live reconstruction updates returned by a connected GPU workstation.

The Unity-based Quest application handles camera capture, head-motion checks, WebSocket communication, runtime loading and rendering of Gaussian splats, ray-based segment selection, and local storage of exported assets. Tracking-aware visibility, opacity pruning, fixed foveated rendering, and a 250,000-splat default cap keep the experience practical on standalone hardware.

The Python backend performs semantic segmentation with GroundingDINO and SAM, initializes sparse geometry through SIFT matching and triangulation, and optimizes Gaussians in PyTorch using gsplat. Semantically labeled splats are clustered into selectable object components with Open3D. Selected components are converted through Screened Poisson reconstruction into a binary PLY mesh with a generated UV atlas and PNG texture.

The result is an end-to-end proof of concept: captured frames drive an evolving reconstruction that returns to the headset for live display, spatial inspection, semantic selection, and textured-mesh export. It was developed as a university group project.