No Pinhole Proxies

01The gap鸿沟

Looks right, measures wrong看起来对，量起来不对

Handheld scanners, backpack and vehicle mapping systems and 360° rigs all pair wide-angle cameras with a LiDAR. Their users want a digital twin that is both: something you can walk through, and something you can put a tape measure on.

手持扫描仪、背包和车载测绘系统、360° 采集设备，都把广角相机和激光雷达装在一起。用户想要的数字孪生要同时做到两件事：能在里面“走”，也能拿尺子去“量”。

Gaussian splatting [1] solved the first half. The second half lags behind. On Oxford Spires [2], where a terrestrial laser scanner (TLS) provides independent ground truth at 1 cm, the official LiDAR maps reach an F-score at 10 cm of 0.54–0.85, while a radiance field trained on the same captures reaches 0.31–0.68.

高斯泼溅 [1] 解决了前一半，后一半还落在后面。在 Oxford Spires [2] 上，地面激光扫描仪（TLS）提供 1 cm 精度的独立真值：官方激光地图的 10 cm F 分数是 0.54–0.85，而用同样数据训练的辐射场只有 0.31–0.68。

The camera side has recently been fixed: native fisheye renderers [3] [4] [5] [6] stopped cutting the fisheye into small perspective images, which threw away field of view and resolution. But a native camera does not change how the LiDAR is taught to the model, and that is where the gap comes from.

相机这一侧最近已经补上了：原生鱼眼渲染器 [3] [4] [5] [6] 不再把鱼眼图切成小块透视图，不再白白丢掉视场和分辨率。但相机原生化并没有改变激光是怎么“教”给模型的，而鸿沟正出在这里。

02Why成因

Two stand-ins that disagree about depth两个替身，对“深度”各说各话

Robot mapping has always treated a sensor reading as a measurement: there is a model of how the sensor works, and the map must agree with it [10] [11]. Radiance fields treat images as pictures to reproduce [12] [1]. With one camera the two views are the same thing. Add a LiDAR and they part ways: inverse rendering is not inverse measurement.

机器人建图一直把传感器读数当作测量：有一个描述传感器如何工作的模型，地图必须与之相符 [10] [11]。辐射场则把图像当作要复现的画面 [12] [1]。只有一台相机时，两种看法是一回事；加上激光雷达，它们就分道扬镳了：逆渲染不等于逆测量。

The common way to bring a LiDAR into a Gaussian renderer is to dress both sensors up as pinhole cameras. The fisheye image is cut into perspective crops, rendered with the affine EWA approximation [8] whose error grows away from the image centre [9]. The LiDAR is replaced by a virtual pinhole depth camera at its origin, which renders

把激光雷达接进高斯渲染器，常见做法是把两种传感器都“打扮”成针孔相机：鱼眼图被切成透视小图，用仿射近似的 EWA 投影渲染 [8]，越往边缘误差越大 [9]；激光雷达被换成放在它原点的一台虚拟针孔深度相机，渲染的是

In words: an opacity-weighted average of the depths of Gaussian centres. z_i is the depth of Gaussian i’s centre along the virtual camera’s axis, \alpha_i\mathcal G_i(q) how much of its footprint covers the query, T_i(q) how much light is left in front of it.

用大白话说：高斯中心深度的加权平均。z_i 是第 i 个高斯中心沿虚拟相机光轴的深度，\alpha_i\mathcal G_i(q) 是它的投影覆盖了这个查询多少，T_i(q) 是它前面还剩多少光。

03Pipeline管线

The whole pipeline on one page一张图看懂整条管线

Before the details, here is where every piece sits. Data flows left to right: two sensors become rays, one renderer turns the same Gaussians into colour and depth, two losses compare them with what was measured, and the update flows back. Switch to “Stand-ins” to see the common approach it replaces.

进入细节之前，先看每一块在哪里。数据从左往右流：两种传感器变成射线，一个渲染器把同一批高斯算成颜色和深度，两个损失分别与实测比较，更新再流回高斯。切到“替身做法”可以看到它所替换的常见做法。

Where this pipeline sits这条管线的位置

The pipeline makes two choices: native rays for the camera, and letting the laser restrict the optimiser. Those two choices also place it among concurrent work. Each contribution below discusses its neighbours in detail.

这条管线做了两个选择：相机用原生射线，激光用来约束优化器。这两个选择也决定了它在同期工作中的位置。下面每个贡献会分别详细讨论各自的近邻工作。

04Contribution 1 · One ray model贡献 1 · 统一射线模型

A pixel and a laser beam are the same thing: a ray像素和激光束本是同一种东西：射线

In one sentence. Stop dressing the sensors up as pinhole cameras. A fisheye pixel is a ray leaving the camera; a laser return is a ray leaving the scanner. Render both the same way, and let both stop at the same place.

一句话。别再把传感器打扮成针孔相机。一个鱼眼像素是从相机出发的一条射线，一个激光回波是从扫描仪出发的一条射线。用同一种方式渲染它们，并让它们停在同一个地方。

\pi_c is the calibrated fisheye model. The whole image circle is used at native resolution, never resampled; there is no crop and no virtual depth camera.

\pi_c 是标定好的鱼眼模型。整个成像圆以原生分辨率使用，不重采样；没有裁切，也没有虚拟深度相机。

Where a ray meets a Gaussian射线在哪里遇到一个高斯

One stopping depth for both sensors两种传感器，同一个停止深度

Walk down the ray, keep track of how much light is left (T), and stop at the first Gaussian after which less than half remains. Where it peaks is the ray’s stopping depth \tau_{\mathrm{term}}. A camera pixel and a laser beam both use this one number: the camera’s depth, the laser’s predicted distance, the surface used for evaluation and the bookkeeping of C2 become one and the same.

沿射线往前走，记下还剩多少光（T），在剩余光第一次少于一半的那个高斯处停下。它的峰值位置就是这条射线的停止深度 \tau_{\mathrm{term}}。相机像素和激光束都用这同一个数：相机的深度、激光的预测距离、评估用的表面、C2 的记账，变成了同一个数。

A push along the beam一股沿光束的推力

The right-hand identity is the intuition: slide the stopping Gaussian one centimetre along the beam and the stopping depth slides one centimetre. A distance error now moves geometry along the line of sight, directly where it was measured.

右边这个恒等式就是直觉：把停止处的高斯沿光束挪 1 cm，停止深度也挪 1 cm。距离误差于是能沿视线方向、在被测量的地方直接推动几何。

What the laser is asked to explain要求激光解释什么

The first term (a robust Huber penalty) asks the beam to stop where it actually returned. The second asks the space the laser just flew through to be empty, up to the sensor’s noise \kappa\sigma_\rho. The camera keeps its usual colour loss and needs no depth loss of its own: the laser already shapes the surface the camera sees.

第一项（稳健的 Huber 惩罚）要求光束停在它真正返回的地方。第二项要求激光刚刚穿过的那段空间是空的（留出传感器噪声 \kappa\sigma_\rho 的余量）。相机保留通常的颜色损失，不再需要自己的深度损失：相机看到的表面，已经由激光塑造好了。

Related work相关工作

Native fisheye rendering. Fisheye-GS [3] adds an equidistant Jacobian to EWA splatting, 3DGUT [4] projects sigma points through the unscented transform, and DirectFisheye-GS [5] builds a Kannala–Brandt model into the rasteriser with cross-view optimisation. 3DGRT [7] evaluates the peak response of a Gaussian along a ray, and 3DGEER [6] shows this response is the exact ray integral and renders it without a BVH. C1’s camera half builds directly on this line. These methods model one sensor and are evaluated on images; C1 brings the LiDAR in as the same kind of ray.

原生鱼眼渲染。Fisheye-GS [3] 给 EWA 泼溅加上等距投影的雅可比，3DGUT [4] 用无迹变换投影 sigma 点，DirectFisheye-GS [5] 把 Kannala–Brandt 模型写进光栅器并加入跨视角联合优化。3DGRT [7] 计算高斯沿射线的峰值响应，3DGEER [6] 证明这正是精确的射线积分，并且不用 BVH 就能渲染。C1 的相机一半直接建立在这条线上。这些方法只建模一种传感器、只在图像上评估；C1 把激光作为同一种射线接进来。

LiDAR as a measurement. DS-NeRF [21] and Urban Radiance Fields [22] treat depth and LiDAR as measurements along rays, including the empty-space term C1 adopts. For splatting, LiDAR-assisted methods such as LI-GS [33] initialise from the point cloud and add a projected depth loss, and GS-LiDAR [19] renders the LiDAR itself through panoramic ray–splat intersections.

把激光当测量。DS-NeRF [21] 和 Urban Radiance Fields [22] 把深度和激光当作沿射线的测量，其中包括 C1 采用的空域项。在高斯泼溅里，LI-GS [33] 等激光辅助方法用点云初始化、再加投影深度损失；GS-LiDAR [19] 则通过全景射线–splat 求交渲染激光本身。

Rendering both sensors. Driving simulators render cameras and LiDAR together but keep them apart. SplatAD [13] gives the LiDAR its own spherical rasteriser, with expected range in training and median range at inference; Uni-Gaussians [14] rasterises cameras and ray-traces LiDAR; XSIM [15] gives each Gaussian one opacity per sensor; UniGaussian [16] shares Gaussians across pinhole and fisheye cameras and uses LiDAR only as supervision. SimULi [17], the closest work, supports fisheye cameras and arbitrary LiDARs in one framework, observes that one set of Gaussians trades image quality against LiDAR accuracy, and therefore splits the scene into two sets. C1 keeps one set and goes after the cause of the conflict instead.

同时渲染两种传感器。驾驶仿真方法会同时渲染相机和激光，但把二者分开处理。SplatAD [13] 给激光单独写了一条球面光栅，训练用期望距离、推理用中位距离；Uni-Gaussians [14] 相机走光栅、激光走光线追踪；XSIM [15] 给每个高斯每种传感器各一个不透明度；UniGaussian [16] 让针孔与鱼眼相机共用高斯，激光只作监督。最接近的 SimULi [17] 在一个框架里支持鱼眼相机和任意激光雷达，观察到一套高斯会在画质和激光精度之间此消彼长，于是把场景拆成两套。C1 则保留一套高斯，转而针对矛盾的来源。

Which depth. Three depth statistics are in use: the opacity-weighted expectation (3DGS; SplatAD in training), the blended plane distance of PGSR [23], and the median where light falls to one half (2DGS [18], GS-LiDAR [19], formalised by Geometry-Grounded GS [20]). C1 adopts the median and makes it the single depth both sensors supervise.

用哪个深度。常用的深度统计量有三种：不透明度加权的期望（3DGS；SplatAD 训练时）、PGSR [23] 混合的平面距离、光量降到一半处的中位深度（2DGS [18]、GS-LiDAR [19]，Geometry-Grounded GS [20] 给出了形式化）。C1 采用中位深度，并让它成为两种传感器共同监督的唯一深度。

05Contribution 2 · Laser-anchored optimisation贡献 2 · 激光锚定优化

Where the laser has measured, images may not move the surface激光量过的地方，图像不能把表面挪走

In one sentence. Do not argue with a measurement by adding yet another weighted loss. Where the laser has seen a surface, let images slide it along but not pull it off; where the laser saw nothing, let images do as they please.

一句话。不要靠再加一个带权重的损失去和测量讨价还价。激光看到过表面的地方，图像可以让它沿表面滑动，但不能把它拉离；激光没看到的地方，图像想怎么改就怎么改。

The standard way to give a radiance field better geometry is to add a loss with a weight. A weight only chooses a point on a trade-off, for a structural reason:

让辐射场几何更好，标准做法是加一个带权重的损失。权重只能在此消彼长的曲线上挑一个点，这有结构上的原因：

For a fixed weight \lambda, the answer is where a straight line of slope -\lambda first touches the set of achievable models. Sweeping \lambda rolls the line around that set, so it can only ever touch its convex outline; any stretch of the best-possible front that bulges outward is unreachable at every weight [25]. A constraint (f_{\mathrm{geo}}\le\varepsilon) has no such blind spot. And a loss never takes a direction away: images can still buy appearance by moving a measured surface.

权重 \lambda 固定时，解落在斜率为 -\lambda 的直线第一次碰到“可达模型集合”的地方。扫一遍 \lambda，就是让这条直线绕着集合滚一圈，所以它只能碰到集合的凸轮廓；最优前沿上任何向外鼓出的一段，不管权重取多少都够不到 [25]。约束（f_{\mathrm{geo}}\le\varepsilon）没有这个盲区。而且损失从不拿走任何一个方向：图像依然可以靠挪动一个已测表面来换取更好的画面。

Bookkeeping along the same rays沿同一批射线记账

Because both sensors now reach the same Gaussians through the same stopping depth, the laser pass can keep a simple record for every Gaussian. For each beam with a return \rho:

两种传感器现在通过同一个停止深度落到同一批高斯上，所以激光那一趟渲染可以顺手给每个高斯记一本简单的账。对每条有回波 \rho 的光束：

• Hit h_i: this Gaussian is where the beam stopped and returned.
• Blocked f_i: it sits in space the beam flew through, weighted by how much light it took.
• Seen n_i: the beam reached it at all.

• 命中 h_i：光束就停在这个高斯上并返回。
• 挡路 f_i：它待在光束飞过的空间里，按它挡掉的光加权。
• 照到 n_i：光束到达过它。

Every 100 steps a running average of the three counts gives each Gaussian one of three labels:

每 100 步，用三个计数的滑动平均给每个高斯贴上三种标签之一：

The anchoring rule锚定规则

For an anchored Gaussian with surface normal \bn_i, the image-driven move \Delta\bmu_i^{\mathrm{rgb}} keeps only its part along the surface; I-\bn\bn^\top is exactly “drop the part through the surface”. The laser’s own update is never touched. Its thickness is held to the sensor noise (\sigma_\rho\approx 3 cm for the Hesai QT64) and it is split only within the surface. A blocking Gaussian fades and is removed, never copied. An unseen Gaussian is optimised exactly as in 3DGS.

对一个法向为 \bn_i 的锚定高斯，图像驱动的位移 \Delta\bmu_i^{\mathrm{rgb}} 只保留沿表面的部分；I-\bn\bn^\top 做的恰好就是“去掉穿过表面的那部分”。激光自己的更新从不被动。它的厚度被压在传感器噪声以内（Hesai QT64 的 \sigma_\rho\approx 3 cm），分裂也只在表面内进行。挡路的高斯逐渐变淡并被删除，绝不复制。未测到的高斯完全按 3DGS 照常优化。

Related work相关工作

Geometry as a weighted loss. Distortion losses [24], normal and planarity terms in 2DGS [18] and PGSR [23], and the depth and line-of-sight terms of DS-NeRF [21] and Urban Radiance Fields [22] all enter as weighted losses. A weighted sum can only reach the convex part of the trade-off [25]; C2 replaces the weight with a rule on what may move.

把几何写成加权损失。distortion 损失 [24]、2DGS [18] 与 PGSR [23] 的法向与平面项、DS-NeRF [21] 与 Urban Radiance Fields [22] 的深度和视线项，都以加权损失的形式进入。加权和只能到达此消彼长曲线的凸部分 [25]；C2 用“什么可以动”的规则取代了权重。

LiDAR restricting the optimiser. EnerGS [26] is the closest work: it also labels LiDAR evidence in three states (occupied, free, unknown) and also takes Gaussian positions away from the image loss, moving them with a geometric energy instead. It does so for every Gaussian, everywhere, from a voxel grid fixed at initialisation; where the laser is silent, positions receive only a weak attractor. C2 takes away one direction, only on Gaussians the laser actually measured, from counts that follow the training, so regions the laser cannot see are still completed from images.

让激光约束优化器。最接近的是 EnerGS [26]：它同样把激光证据分成三类（占据、空闲、未知），同样把高斯位置从图像损失手里拿走，改由几何能量来移动。但它作用于所有高斯、所有地方，依据的是初始化时就固定的体素网格；激光没测到的地方，位置只剩很弱的吸引项。C2 只拿走一个方向、只作用于激光真正测到的高斯，依据的计数随训练更新，所以激光看不到的地方仍能靠图像补全。

Other constraints. GeoP-Calib [27] stops image gradients to all Gaussians while calibrating camera–LiDAR extrinsics; Peng et al. [28] constrain camera parameters with barriers; the cross-view optimisation of DirectFisheye-GS [5] restrains Gaussians with photometric agreement between views. C2 acts per Gaussian and per direction, and its signal is a measurement rather than another photometric cue.

其他约束方式。GeoP-Calib [27] 在标定相机–激光外参时，阻断图像对所有高斯的梯度；Peng 等 [28] 用障碍函数约束相机参数；DirectFisheye-GS 的跨视角优化 [5] 用视角间的光度一致性约束高斯。C2 按单个高斯、单个方向起作用，信号来自测量，而不是又一个光度线索。

06Contribution 3 · Speed贡献 3 · 速度

More faithful, without slowing down更忠实，但不变慢

In one sentence. Keep every trick that makes the existing trainer fast, and change only the one formula that turns a Gaussian into opacity.

一句话。保留现有训练器所有让它快的手段，只换掉“把高斯变成不透明度”的那一个公式。

Our trainer carries the rasterisation lineage’s accelerations: GPU-resident storage, tile binning with warp-cooperative duplication, radix sorting, an Adam that updates only visible Gaussians, fused SSIM and a warp-shuffle backward pass. C1 and C2 keep every one of them.

我们的训练器带着光栅化这条线的各种加速：数据常驻 GPU、按 tile 分箱并由 warp 协作复制、基数排序、只更新可见高斯的 Adam、融合 SSIM、warp-shuffle 反向。C1 和 C2 把它们全部保留。

Why the exact response is cheap为什么精确响应不贵

Per pixel–Gaussian pair, two dot products and one division remain. With the six products d_jd_k of a pixel’s direction formed once outside the loop, q is six fused multiply-adds. The main thing that grows is the number of pixels used: a native Oxford fisheye frame has about 2.9× the pixels of the official pinhole crop, all of it field of view the crop would have discarded.

每个“像素×高斯”对，只剩两个点积和一次除法。像素方向的六个乘积 d_jd_k 在循环外算好之后，q 就是六次乘加。真正增长的是用到的像素数：Oxford 一帧原生鱼眼的像素约是官方针孔裁切的 2.9 倍，多出来的全是裁切本会丢掉的视场。

Related work相关工作

Faster rasterisation. Taming 3DGS [29] bounds the Gaussian budget and introduced the sparse, visible-only Adam our trainer inherits; Faster-GS [30] restructures the backward pass and fuses the optimiser; StopThePop [31] addresses the per-Gaussian sorting that exact responses still rely on. C3 carries these gains over to rays unchanged.

更快的光栅化。Taming 3DGS [29] 控制高斯预算，并提出了我们训练器继承的“只更新可见高斯”的稀疏 Adam；Faster-GS [30] 重构反向并融合优化器；StopThePop [31] 处理精确响应仍依赖的逐高斯排序。C3 把这些收益原样带到射线上。

Exact rays, and two sensors. 3DGRT [7] obtains exact responses by tracing a BVH; 3DGEER [6] renders the same response without one, near rasterisation speed. SplatAD [13] gives the LiDAR a second, spherical rasteriser. C3 puts the exact response inside the existing tile rasteriser and lets the LiDAR reuse the camera’s binning and formula.

精确射线与两种传感器。3DGRT [7] 通过遍历 BVH 得到精确响应；3DGEER [6] 不用 BVH 也能渲染同样的响应，速度接近光栅化。SplatAD [13] 为激光另写了一条球面光栅。C3 把精确响应放进现有的 tile 光栅器，并让激光复用相机的分箱和公式。

07Experiments实验

Four questions the experiments answer实验要回答的四个问题

Evaluation is built on Oxford Spires Keble-04 [2], to our knowledge the only public dataset with fisheye colour cameras (three 1440 × 1080), a synchronised mobile LiDAR (Hesai QT64) and independent survey geometry (TLS at 1 cm). Every method trains on one RTX 5090 for the same wall-clock time with three seeds, and is scored the same way.

评测建立在 Oxford Spires Keble-04 [2] 上。据我们所知，它是唯一同时具备鱼眼彩色相机（三台 1440 × 1080）、同步移动激光雷达（Hesai QT64）和独立测绘几何（1 cm TLS）的公开数据集。每个方法都在同一张 RTX 5090 上、相同墙钟时间、三个随机种子下训练，并用同一把尺子打分。

08Explore scenes浏览场景

Step inside a Gaussian scene走进一个高斯场景

Choose a published example, or open a Gaussian PLY from your device. It is rendered in your browser; local files are never uploaded.

选择一个已发布的示例，或从本机打开一个高斯 PLY。它在你的浏览器里渲染，本地文件不会上传。