Specular-Diffuse Disentanglement
in Neural Rendering

Personal Project

Project Overview

The core objective of my project is to achieve separation of diffuse and specular color components in neural rendering. While models like Ref-NeRF have demonstrated success in this area through the use of directional embedding and structured model separation, their reliance on Mip-NeRF results in slow performance. To address this, I implemented key elements of Ref-NeRF using the Insatnt-NGP-based NeRFacto model as a foundation (presented by NeRFStudio), enabling faster diffuse-specular disentanglement in an accelerated NeRF system.


Key Structures

Reflection Direction Parameterization

Ref-NeRF reparameterizes outgoing radiance in terms of the reflection direction

$$ \hat{\omega}_r = 2(\hat{\omega}_o \cdot \hat{n}) \hat{n} - \hat{\omega}_o, $$

where $\hat{\omega}_o$ is the viewing direction and $\hat{n}$ is the surface normal.

This approach enhances the interpolation of specular reflections by focusing on the reflected view direction rather than the direct one.


Integrated Directional Encoding (IDE)

Drawing inspiration from Mip-NeRF, IDE employs spherical harmonics and a von Mises-Fisher (vMF) distribution to encode reflectance vectors. This method allows the model to efficiently handle varying material roughness. The expected spherical harmonics for the vMF distribution are approximated by:

$$ E_{\hat{\omega} \sim \text{vMF}(\hat{\omega}r, \kappa)}[Y^m_{\ell}(\hat{\omega})] \approx A_{\ell}(\kappa) Y^m_{\ell}(\hat{\omega}r) $$

where $ A_{\ell}(\kappa) $ is the attenuation function based on roughness $\rho$.


Diffuse and Specular Color Decomposition

Radiance is divided into diffuse ($c_d$) and specular ($c_s$) components, combined as:

$$ c = \gamma(c_d + \gamma_s \odot c_s) $$

where $\gamma$ is a color transformation to sRGB, ensuring the final output falls within the $[0, 1]$ range.


Implementation

Building on NeRFStudio's Nerfacto model, I integrated Ref-NeRF's directional embedding and model structure to enable diffuse-specular disentanglement. By leveraging Instant-NGP, I achieved significantly faster performance, addressing the speed limitations of Mip-NeRF-based approaches while maintaining a high-quality rendering of specular highlights and reflections.

Furthermore, implementing this project within NeRFStudio allowed for efficient rendering of scenes with dynamic camera paths, enhancing both the visual output and interaction with 3D scenes.

Below are my results for the Mip360 garden and two scenes I personally captured.

Mip-NeRF 360 Garden
Full Diffuse Specular
Personal Capture #1
Full Diffuse Specular
Personal Capture #2
Full Diffuse Specular