Neural Rendering in Game Engine

NC Research

Overview

The project aims to explore the fusion of neural rendering and game engines. We design a pipeline for efficient scene modeling, reconstruction using neural rendering, and subsequent integration of the reconstructed scene into a game engine.

Neural Rendering w/ 360 Videos

Using 360 Cameras for effective large scene capturing

To efficiently capture the large scene for 3D scene reconstruction, we use 360 videos for our GT NVS sources.

Spherical 3D Gaussian Splatting

Initially, we trained the 3D GS model using 360-degree images, but the results were poor. This was likely due to stitching errors between the fisheye images that comprise the equirectangular 360-degree images.

To address this issue, we decided to use the original fisheye images as our ground truth (GT) sources. Furthermore, we implemented a custom CUDA-based Gaussian rasterization module to support the fisheye camera model, which was not supported by the original rasterization module.

Fisheye Camera Model	Spherical Camera Model

3D GS Reconstruction

Here is the final reconstructed scene from 360 camera capture.

Gaussian Splatting w/ Game Engine

With the reconstructed 3D neural rendering scene, we further integrate it into game engines such as Unity or Unreal, which offer powerful synthetic world generation capabilities.

The integration of the Gaussian Splatting and Game Engine can be easily implemented using GS rasterization rule. We further optimize it using vector-quantization so that the Unity-GS can be rendered within its original rapid performance.

Below is the GS's virtual world experience which is fully interactive within the Unity engine, with our neural avatar also reconstructed by radiance fields technique.

In Unreal Engine (UE), Gaussian Splatting scenes can be constructed similarly. However, we've encountered a significant limitation: Unreal's Niagara system can effectively render only up to 2 million particles. Given that a fully reconstructed scene often consists of over 6 million particles, a simple copy-and-paste approach leads to disastrous results. This issue has also been reported with the XVERSE's UE GS plugin.

To overcome this, it's necessary to prune the splats so that the particle count stays within UE's upper limit. We define the contribution of each splat as the sum of the intersected rays across all the training images. Mathematically, this can be expressed as:

$$ C = \sum_{k=1}^{n} C_k, \quad C_k = \sum_{p \in \mathcal{P}_k} \alpha_i(p) \prod_{j=1}^{i(p)-1} (1 - \alpha_j) $$

We estimated the contribution of all trained splats and pruned the particles accordingly. The pruned GS scene was then aligned. This prune-and-refine process was iteratively optimized through only a few steps. Below is a comparison between the original GS scene and the pruned GS scene in UE.

Original GS in UE	Pruned GS in UE

Below is interactive result in UE.

Please see also my related blog post: Can NeRF be Used in Game?