SA-GS

Under zoom-in, 3D Gaussian Splatting (3DGS) exhibits significant erosion artefacts, while under zoom-out, it undergoes dramatic dilation. Mip-Splatting utilizes 3D smoothing and 2D Mip filters to regularize primitives during training. In contrast, our method is training-free and maintains scale consistency using solely a single 2D scale-adaptive filter. Scale adaptation allows us to use super-sampling (named as SA-GS_sup in the future) and its limiting case integration (named as SA-GS_int in the future) to obtain more accurate results when zooming out.

Abstract

In this paper, we present a Scale-adaptive method for Anti-aliasing Gaussian Splatting(SA-GS). While the state-of-the-art method Mip-Splatting needs modifying the training procedure of Gaussian splatting, our method functions at test-time and is training-free. Specifically, SA-GS can be applied to any pretrained Gaussian splatting field as a plugin to significantly improve the field's anti-alising performance. The core technique is to apply 2D scale-adaptive filters to each Gaussian during test time. As pointed out by Mip-Splatting, observing Gaussians at different frequencies readily leads to mismatches between the Gaussian scales during training and testing. Mip-Splatting resolves this issue using 3D smoothing and 2D Mip filters, which are unfortunately not aware of testing frequency. In this work, we show that a 2D scale-adaptive filter that is informed of testing frequency can effectively match the Gaussian scale, thus making the Gaussian field remain distributed consistently across different testing frequencies. When scale inconsistency is eliminated, sampling rates smaller than the scene frequency result in conventional jaggedness, and we propose to integrate the projected 2D Gaussian within each pixel during testing. This integration is actually a limiting case of super-sampling, which significantly improves anti-aliasing performance over vanilla Gaussian Splatting. Through extensive experiments using various settings and both bounded and unbounded scenes, we show SA-GS performs comparably with or better than Mip-Splatting. Note that super-sampling and integration are only effective when our scale-adaptive filtering is activated.

Motivation & Methods

Paradigm Comparison of Gaussian Rasterization Process. All Gaussian Splatting methods share this framework for training and rendering, but different models use different strategies to process Gaussian primitives. Both 3DGS and Mip-Splatting suffer from scale inconsistency and need to modify the training procedure. Our approach is training-free and only operates on the testing flow. We use (d) in pixel space to maintain the scale consistency of the Gaussian primitives, and further enhance the anti-aliasing capability of 3DGS by applying (e) and (f) to the α-blending process. Note that (e) and (f) only make sense with (d) activated.

Scale ambiguity & 2D Scale-adaptive Filter. The heuristic 2D dilation process in vanilla 3DGS code operates on the pixel space and enlarges the projected 2D Gaus- sian by a fixed amount (around 1.64 pixel). However, a fixed 2D dilation (1.64 pixel) can result in scale ambiguities when representing the same scene at different rendering settings, as shown by the green expansion area. (a) When the Gaussian scale is held constant and the resolution changes, the dilation scale (green) changes inconsistently. (b) When the Gaussian scale changes and the resolution remains constant, the dilation scale (green) does not change with the Gaussian. Our 2D scale-adaptive filter ensures that the Gaussian scale remains consistent across different rendering settings, as shown by the red expansion area. This keeps the scale consistent with the training setup.

Super Sampling and Integration applied on Gaussian. Our super sampling method, denoted as (a), involves dividing each pixel thread into 9 sub-pixels when traversing the base-ordered Gaussian within a tile. Each sub-pixel independently undergoes α-blending and weights the Gaussian spherical harmonic coefficient according to the sampling results. (b) is our integration method that diagonalizes the Gaussian covariance matrix by pixel rotation. This decomposes the integration result into the product of two marginal Gaussian distributions.

Results

Comparison on Mip-NeRF 360 Dataset

We compare our method with Mip-Splatting and 3DGS in the zoom-out (first row) and zoom-in (second row) cases, respectively. Our method eliminates the dilation & erosion artefacts of 3DGS under different rendering settings, while improving the jaggedness effect in the zoom-out case. Our method obtains results comparable with and beyond (zoom-out case) Mip-Splatting.

Comparison on Blender Dataset

We show both zoom-out and zoom-in cases in different scenarios. We highlight the contrasting areas. Our method obtains robust anti-aliasing performance improvements over 3DGS, while significantly outperforming Mip-Splatting in the zoom-out case.

Effectiveness of 2D Scale-adaptive Filter

The 2D scale-adaptive filter maintains the consistency of the Gaussian distribution when zooming out, which in turn fully unleashes the power of the integral and supersampling antialiasing methods. Additionally, the filter removes erosion artifacts from the scene when zooming in, resulting in a more structurally uniform scene.

BibTeX

@article{song2024sa,
      title={SA-GS: Scale-Adaptive Gaussian Splatting for Training-Free Anti-Aliasing},
      author={Song, Xiaowei and Zheng, Jv and Yuan, Shiran and Gao, Huan-ang and Zhao, Jingwei and He, Xiang and Gu, Weihao and Zhao, Hao},
      journal={arXiv preprint arXiv:2403.19615},
      year={2024}
    }