pixelSplat Under Scrutiny: Does Probabilistic Depth Sampling Genuinely Resolve Scale Ambiguity, or Inherit It from RealEstate10K?

Abstract

pixelSplat [Charatan et al. 2024; arXiv:2312.12337] proposes a feed-forward network that ingests two posed RGB images and emits a set of 3D Gaussians, enabling novel view synthesis without per-scene optimization. The central methodological claim is that scale ambiguity, notorious for inducing local minima in two-view depth regression, is sidestepped through a probabilistic depth parameterization: rather than regressing a scalar depth per pixel, the network predicts a categorical distribution over discrete depth bins along the epipolar line and samples Gaussian means from it using a reparameterization-style gradient. The authors report strong PSNR gains over PixelNeRF [Yu et al. 2021] and GPNR [Suhail et al. 2022] on RealEstate10K and ACID, alongside rendering that is one to two orders of magnitude faster than NeRF-family generalizable baselines. My read is that the paper is a genuinely clever engineering advance for two-view generalizable radiance fields, but the scale-ambiguity story is considerably weaker than advertised: the probabilistic sampler does not resolve scale ambiguity so much as outsource it to a dataset prior, and the epipolar transformer likely memorizes the layout statistics of indoor walkthroughs. The qualitative results are more revealing than the numbers.

Key Contributions

The paper stakes out three contributions. First, it casts feed-forward 3D Gaussian Splatting as a two-view inference problem and demonstrates that scene parameter prediction can be amortized across a dataset of posed video clips. Second, it introduces a probabilistic depth parameterization intended to render trainable, via straight-through gradients, the discontinuous step of placing a Gaussian at a specific depth. Third, it argues that this parameterization avoids the mode-collapsed, over-smoothed reconstructions that plague end-to-end two-view depth regression under scale ambiguity.

I would classify the contribution as moderate, trending toward significant on engineering grounds but incremental on theoretical grounds. The architectural skeleton is familiar: epipolar attention has been with us since GPNR and IBRNet [Wang et al. 2021], and per-pixel prediction of scene primitives was already the operating principle of PixelNeRF. What is new is (i) emitting 3DGS primitives rather than radiance-field samples, which delivers the rasterization-speed payoff of Kerbl et al. [2023], and (ii) the probabilistic depth trick. The second deserves closer auditing.

Methodology

Let $I_1,I_2\in {\mathbb{R}}^{H\times W\times 3}$ be two input views with known intrinsics $K_i$ and relative pose $(R,t)$. For each pixel $p$ in the reference view, pixelSplat predicts a 3D Gaussian ${\mathcal{G}}_p=({\mu }_p,{\Sigma }_p,c_p,{\alpha }_p)$, where ${\mu }_p\in {\mathbb{R}}^3$ is the Gaussian mean, ${\Sigma }_p$ is a covariance parameterized by scale and rotation, $c_p$ are spherical harmonic coefficients, and ${\alpha }_p$ is opacity. The mean is the contested quantity.

Rather than regressing ${\mu }_p=o+d_p\cdot r_p$ with $d_p$ a scalar depth and $r_p$ the ray direction, the authors discretize depth into $B$ bins along the ray, predict a logit vector ${\ell }_p\in {\mathbb{R}}^B$, and sample

Gradients flow through the sampling step via a straight-through estimator, effectively allowing the network to shift probability mass between bins. Multiview features are aggregated by an epipolar transformer: for each pixel in the reference image, the network attends along the corresponding epipolar line in the second view, yielding a feature that is decoded into ${\ell }_p$ and the remaining Gaussian parameters.

Training uses posed video clips from RealEstate10K [Zhou et al. 2018] (indoor real-estate walkthroughs) and ACID [Liu et al. 2021] (aerial coastlines), with two frames sampled as input and a third as supervision. The loss is a photometric rendering loss (L2 plus LPIPS); no explicit geometric loss is imposed. The rendered Gaussians are passed through the differentiable rasterizer of Kerbl et al. [2023].

Architecturally, the backbone is a DINO-ViT-B [Caron et al. 2021] encoder per image, followed by cross-image epipolar attention. Depth bins are chosen via a disparity-uniform schedule with $B=128$ bins, with near and far planes at approximately $1$ and $100$ in scene units, where "scene units" are the SfM-derived scales of the training data.

That last parenthetical is where the critique begins.

Results & Analysis

The headline numbers, reconstructed from the paper's tables:

The absolute gains over AttnRend are roughly 1.1 PSNR and 0.07 LPIPS on RE10K, with render time dropping by approximately $16\times$. On ACID the PSNR gap is 1.4. These are meaningful improvements, but not the kind of transformative delta that forces a rethink of the field.

The more interesting result is the ablation pitting probabilistic against deterministic depth. The authors report that replacing the categorical sampler with direct depth regression collapses PSNR by roughly 3 dB and produces visually blurry, fog-like reconstructions. This is the central empirical argument for the design. I find it convincing at face value but incomplete: we do not see the obvious intermediate, namely, a soft expected-depth parameterization ${\mu }_p=o+r_p{\sum }_b\mathrm{softmax}({\ell }_p{)}_b{d}_b$ without sampling. This is the standard mixture-of-depths trick, and it sidesteps both the regression collapse and the sampling variance. Its omission is precisely the kind of missing ablation that, in a CVPR Area Chair review, I would ask for explicitly.

Statistical significance is not reported. PSNR differences of this magnitude on 2,500-clip test splits are almost certainly outside noise, but there are no confidence intervals, no seeds, no per-scene breakdowns.

The Weakest Link: Scale Ambiguity Is Not Resolved, It Is Cached

The paper frames scale ambiguity as a gradient-flow problem: given two posed images, scene depth is ambiguous up to a scalar when pose comes from SfM and the SfM scale is arbitrary. The probabilistic sampler is offered as the cure. This framing misdirects.

The training data, RE10K and ACID, has scene scale fixed by the SfM reconstruction pipeline that produced the camera poses. That pipeline assigns a scale per clip, typically by normalizing such that some reference distance is unit. The network sees consistent "scene units" across training because all clips were processed by the same pipeline under the same normalization convention. At test time, so long as the query poses come from the same pipeline, the network can assume the same prior scale. It does not resolve scale ambiguity; it memorizes a dataset-level convention.

This is not merely a philosophical point. It has a concrete empirical test: take a trained pixelSplat model and feed it poses where the translation $t$ has been multiplied by an arbitrary factor $\lambda \in \{0.5,2,5\}$. Under genuine scale invariance, the output geometry should scale correspondingly and renderings should be unchanged. I predict the renderings degrade sharply. This experiment is not in the paper. It would take two hours to run.

The probabilistic sampler does solve a real optimization problem, namely, that placing a Gaussian at a depth $d$ is a discrete decision, and L2 photometric loss around the wrong depth yields uninformative gradients. But the same problem is solved by expected-depth parameterization, by coarse-to-fine bin refinement [as in MVSNet, Yao et al. 2018], or by unprojection-and-weighted-sum as in IBRNet. Framing the categorical sampler as a resolution of scale ambiguity conflates two independent phenomena.

Alternative Interpretation: An Epipolar Prior Memorizer with a Gaussian Decoder

Here is a reading of the same evidence that the authors do not consider. The epipolar transformer, trained on hundreds of thousands of RE10K clips, is learning a very strong prior over indoor scene layouts: the camera is near eye height, the floor lies below, walls are vertical, furniture occupies mid-depth regions. Given two views with known pose, epipolar geometry constrains where a pixel's match *could* be; the learned prior selects *where it probably is* from among those possibilities.

Under this reading, pixelSplat is not doing two-view stereo with a clever depth parameterization. It is doing monocular-ish depth prediction regularized by epipolar feasibility, with the Gaussian decoder handling the rendering. The probabilistic sampler helps because it lets the network express uncertainty over *which* feasible match is correct and commit only when the prior concentrates.

Two pieces of evidence support this alternative. First, pixelSplat's performance on ACID, whose scene statistics differ from RE10K's, is relatively weaker than on RE10K despite the physics of two-view stereo being identical. If the method were genuinely solving geometric correspondence, dataset-shift penalties would be smaller. Second, the method struggles with wide baselines, exactly what one would expect if the learned prior is calibrated to RE10K's modest baseline distribution.

A fair comparison, which the paper does not make, is against a method that explicitly separates geometric correspondence from scene prior. DUSt3R [Wang et al. 2024; arXiv:2312.14132] regresses pointmaps directly and does not assume known pose. MVSplat [Chen et al. 2024] uses cost-volume stereo with Gaussian decoding and offers a more apples-to-apples baseline for the "does the probabilistic sampler actually help" question. MVSplat reports comparable RE10K PSNR with cost-volume-based depth, suggesting the sampler is not the unique enabler.

Assumption Audit

Three implicit assumptions deserve surfacing.

Assumption 1: Input pose is accurate. pixelSplat takes poses as ground truth. In real deployments, pose comes from VIO, COLMAP, or learned pose estimators, each with error distributions the network has never seen. Pose error and depth prediction interact nonlinearly: a 2-degree rotation error around a pixel at depth $d$ translates to a lateral shift of approximately $d\cdot \sin (2°)\approx 0.035d$, which at $d=10\mathrm{m}$ is 35 cm. A Gaussian placed with that lateral error will render with visible ghosting. The paper does not test robustness to pose noise.

Assumption 2: Training pose scale transfers to inference. As argued above.

Assumption 3: Photometric loss is sufficient supervision for geometry. Without explicit geometric regularization (depth smoothness, surface normal consistency, multi-view stereo losses), infinitely many scene geometries render identically from two views. The network picks one using its prior. For novel-view synthesis from a third pose, this is often adequate; for downstream tasks like robotics or AR that demand metrically faithful geometry, it is not.

Limitations and Failure Modes

Beyond what the authors acknowledge, I see four concrete failure modes.

Thin structures and transparent objects. Per-pixel Gaussians are a surface representation. A chain-link fence or a wine glass breaks the assumption of one Gaussian per ray. The paper's qualitative results conspicuously avoid these cases.

Specular and non-Lambertian surfaces. The photometric loss assumes appearance is view-consistent enough that multi-view disagreement signals geometric error. For mirrors and polished floors, common in RE10K interiors, the assumption fails, and the network hallucinates plausible but geometrically wrong surfaces behind the mirror.

Wide-baseline configurations. When the baseline is much larger than RE10K's typical, the epipolar line becomes long and the categorical prior over $B=128$ bins grows diffuse. I would predict a sharp drop beyond roughly twice the training baseline distribution.

Cross-dataset transfer. RE10K has a very specific capture distribution (handheld video, indoor, eye-height). Applying pixelSplat to ScanNet or to outdoor KITTI-style driving data should expose the prior-memorization hypothesis directly.

Related Work

Three works frame the critique.

DUSt3R [Wang et al. 2024] abandons known pose entirely and regresses pointmaps from image pairs. Its success without a probabilistic sampler is direct evidence that two-view 3D prediction does not require pixelSplat's specific depth parameterization. The more interesting question is whether DUSt3R's pointmap regression and pixelSplat's categorical depth sampling are solving the same problem with different inductive biases, or genuinely different problems.

MVSplat [Chen et al. 2024] swaps the epipolar transformer for a cost volume and achieves similar numbers. Since cost volumes are the classical stereo primitive, MVSplat is the natural ablation for the question: does pixelSplat's performance come from the Gaussian output format or from the epipolar/probabilistic machinery? The evidence favors "mostly the output format."

LatentSplat and Splatter Image [Szymanowicz et al. 2024] push the two-view-to-Gaussians paradigm further with simpler depth parameterizations and competitive results, again suggesting the probabilistic sampler is not the load-bearing component.

What Would Change My Mind

I would revise my assessment upward given any of the following.

1. A scale-invariance ablation: multiply test-time pose translations by $\lambda \ne 1$ and report PSNR as a function of $\lambda$. If PSNR is flat across a decade of $\lambda$, scale ambiguity is genuinely resolved.

2. A cross-dataset generalization result: train on RE10K, test on ScanNet without fine-tuning. A modest PSNR drop (under 3 dB) would indicate the epipolar machinery generalizes beyond the prior.

3. An ablation against expected-depth soft parameterization, holding the epipolar transformer fixed. If probabilistic sampling still wins by 2+ dB, the specific parameterization matters.

4. Results under pose noise: inject Gaussian pose perturbation with ${\sigma }_R=1°$, ${\sigma }_t=5\%$ of scene scale, and report the PSNR degradation curve.

Conversely, my critique would be strengthened by any of the following: (a) worse transfer to ACID than to RE10K despite ACID's simpler scene geometry, (b) catastrophic failure under pose scale perturbation, (c) performance equal to MVSplat once controlled for encoder and training data.

Broader Implications

If pixelSplat's framing is correct, the probabilistic depth parameterization is a general tool that should transfer to any feed-forward scene-primitive prediction problem. If my reading is correct, then the field's recent enthusiasm for "generalizable 3DGS from two views" is really enthusiasm for "amortized scene priors over SfM-normalized datasets," and the correct comparison is not to classical two-view stereo but to single-image scene prediction models conditioned on a second image. The bar for claiming generalization should shift accordingly.

For reviewers: when a paper claims to resolve a well-known ill-posed problem (scale ambiguity, depth from stereo with known pose, and the like), check whether the training data silently resolves it first. Networks trained on pipeline-normalized data are not invariant to the normalization; they are calibrated to it.

Verdict

pixelSplat is a strong engineering contribution to generalizable 3DGS and deserves its citations. The rasterization-speed payoff is real, and the epipolar-to-Gaussian pipeline is a useful blueprint. The paper's theoretical framing, which casts probabilistic depth sampling as a resolution of scale ambiguity, overstates what the method does. The ablation against soft expected-depth is missing, the scale-invariance test is missing, and the cross-dataset generalization evidence is thin. Novelty rating: moderate. The next experiment I would run is the $\lambda$-scale perturbation. That single chart would settle the central interpretive question.

Reproducibility & Sources

Primary paper. Charatan, D.; Li, S.; Tagliasacchi, A.; Sitzmann, V. "pixelSplat: 3D Gaussian Splats from Image Pairs for Scalable Generalizable 3D Reconstruction." CVPR 2024. arXiv:2312.12337.

Code repository. Official code released by the authors at github.com/dcharatan/pixelsplat (release accompanying the CVPR 2024 camera-ready).

Datasets.

• RealEstate10K [Zhou et al. 2018]: publicly available via the RealEstate10K project page; licensed for research.
• ACID [Liu et al. 2021]: publicly available via the Infinite Nature project page; licensed for research.

Reproducibility assessment (1, 5 scale).

• Code availability: 5. Official training and evaluation code is released alongside reference checkpoints.
• Data availability: 4. Both datasets are public, but RE10K requires scraping YouTube URLs, several of which are now dead; the exact train/test splits used in the paper may not be fully reconstructible.
• Experimental detail sufficient: 3. Architecture and hyperparameters are specified, but straight-through estimator details, the bin schedule, and specific LPIPS weighting require code inspection to reproduce exactly.

References (Inline)

[Caron et al. 2021] Caron, M. et al. "Emerging Properties in Self-Supervised Vision Transformers." ICCV 2021.

[Charatan et al. 2024] Charatan, D. et al. "pixelSplat." CVPR 2024. arXiv:2312.12337.

[Chen et al. 2024] Chen, Y. et al. "MVSplat: Efficient 3D Gaussian Splatting from Sparse Multi-View Images." ECCV 2024.

[Du et al. 2023] Du, Y. et al. "Learning to Render Novel Views from Wide-Baseline Stereo Pairs." CVPR 2023.

[Kerbl et al. 2023] Kerbl, B. et al. "3D Gaussian Splatting for Real-Time Radiance Field Rendering." SIGGRAPH 2023.

[Liu et al. 2021] Liu, A. et al. "Infinite Nature: Perpetual View Generation of Natural Scenes." ICCV 2021.

[Mildenhall et al. 2020] Mildenhall, B. et al. "NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis." ECCV 2020.

[Suhail et al. 2022] Suhail, M. et al. "Generalizable Patch-Based Neural Rendering." ECCV 2022.

[Szymanowicz et al. 2024] Szymanowicz, S. et al. "Splatter Image: Ultra-Fast Single-View 3D Reconstruction." CVPR 2024.

[Wang et al. 2021] Wang, Q. et al. "IBRNet: Learning Multi-View Image-Based Rendering." CVPR 2021.

[Wang et al. 2024] Wang, S. et al. "DUSt3R: Geometric 3D Vision Made Easy." CVPR 2024.

[Yao et al. 2018] Yao, Y. et al. "MVSNet: Depth Inference for Unstructured Multi-view Stereo." ECCV 2018.

[Yu et al. 2021] Yu, A. et al. "pixelNeRF: Neural Radiance Fields from One or Few Images." CVPR 2021.

[Zhou et al. 2018] Zhou, T. et al. "Stereo Magnification: Learning View Synthesis using Multiplane Images." SIGGRAPH 2018.