Visual Positioning Systems
The Expert Report

01What a VPS is — and what it replaces

A Visual Positioning System is an infrastructure-plus-algorithm stack that determines a camera's 6-DoF pose — three degrees of position (x, y, z) and three of orientation (roll, pitch, yaw) — by comparing the current camera image against a previously constructed visual model of the environment. The academic name for the online problem is visual localization or camera relocalization; "VPS" is the productized, planet-or-venue-scale version of it.

The single most important framing: a VPS is a lookup against a map, not a sensor. GPS measures signals from satellites; a VPS recognizes place. Everything about VPS engineering — coverage, freshness, storage, privacy — follows from the fact that someone must build and maintain the map before anyone can localize inside it.

Why GPS is not enough

The killer capability is orientation. Augmented reality, robot manipulation, and pedestrian guidance ("the entrance is behind you") all need heading. Magnetometers drift and lie near steel; GPS heading needs motion. A VPS delivers position and a gravity-consistent, drift-free orientation from a single glance at the world.

Relatives and non-relatives

• SLAM (Simultaneous Localization and Mapping) builds a map while tracking in unknown space. A VPS localizes against a map that already exists. Production stacks use both: SLAM/VIO for smooth local tracking, VPS for absolute global anchoring (Section 8).
• Place recognition / image retrieval answers "which place is this?" (coarse, image-level). A VPS goes further to metric pose. Retrieval is the first stage of most VPS pipelines (Section 5).
• Fiducial markers (QR, AprilTag, ArUco) are the poor man's VPS: engineered features with known pose. Cheap, robust, ugly, and require physically instrumenting the space.
• WiFi RTT / BLE beacons / UWB give 1–5 m (WiFi/BLE) or 10–30 cm (UWB) indoor position but no orientation and require deployed radio hardware. UWB is the strongest non-visual competitor for instrumented venues.

The radio cousin — WiFi CSI (Channel State Information)

Every modern WiFi packet crosses a room as an OFDM waveform on dozens-to-thousands of subcarriers per antenna pair. To equalize the link, the receiver estimates the complex channel response — amplitude and phase, per subcarrier. That estimate, the CSI, is a multipath fingerprint of the space: every wall, cabinet, and human body contributes reflections, and any movement reshapes the response packet by packet. Read CSI as a time series and the access point becomes a passive sensor.

• Device-free sensing. Presence and intrusion detection, gesture recognition, fall detection, gait identification, even respiration monitoring — through walls, in darkness, with nothing worn or carried. The tagline is literal: the router can see you move.
• Positioning. CSI fingerprinting reaches decimeter-to-meter accuracy indoors — far beyond coarse RSSI — and time/angle-of-flight methods (WiFi RTT / 802.11mc, MIMO phase) solve position directly. Still no orientation, and still an order coarser than a VPS fix.
• Standardization and accessibility. IEEE 802.11bf is making sensing a first-class WiFi service, and ESP32 CSI toolkits put the capability on $5 microcontrollers — the hobbyist ecosystem around "ESP-CSI" is already large.

For a VPS architect, CSI is a complement, not a competitor. It is the natural upgrade to the coarse radio prior in Section 5's query package — sharper tile selection, reliable floor disambiguation — and it covers exactly the cases vision cannot: lights off, lens occluded, camera-prohibited spaces. The privacy note of Section 10 applies with extra force here: device-free sensing is consent-free by default, and it evades every camera-era intuition about when one is being watched.

02The geometric foundation

Every VPS reduces to one equation: how a 3D world point becomes a 2D pixel. Master this and the rest of the field is bookkeeping.

The pinhole projection model

• X = (X, Y, Z, 1)ᵀ — a point in the world coordinate frame (the map's frame).
• [R | t] — the extrinsics: a 3×4 matrix built from a rotation R ∈ SO(3) and translation t ∈ ℝ³ that transforms world coordinates into camera coordinates. This is the pose. This is the entire output of a VPS. The camera center in world coordinates is C = −Rᵀt — a fact interview questions love.
• K — the intrinsics: focal lengths f_x, f_y and principal point (c_x, c_y) arranged as an upper-triangular 3×3. Real lenses add radial/tangential distortion (Brown–Conrady coefficients k₁, k₂, p₁, p₂), which must be calibrated out. Phones expose calibrated intrinsics through ARKit/ARCore, which is one reason mobile VPS is tractable.

Reprojection error — the universal currency

Given a candidate pose, project every matched 3D point into the image and measure the pixel distance to where the feature was actually observed:

Reprojection error in pixels is the field's shared objective function. Mapping minimizes it over all cameras and points jointly (bundle adjustment); localization minimizes it over one camera's pose with the map fixed.

Coordinate frames — where beginners drown

A production VPS juggles at least four frames: world/map frame (often georegistered: aligned to Earth via GPS priors or surveyed control points, so poses convert to latitude/longitude/altitude + heading), camera frame, the device's VIO/odometry frame (arbitrary origin at session start, drifts over time), and gravity-aligned ENU (East-North-Up) for anything user-facing. The VPS fix is precisely the transform that pins the drifting odometry frame to the world frame. Write the frames down; most integration bugs are frame bugs, and most of those are "inverse of the transform you meant."

The PnP problem

Perspective-n-Point: given n correspondences between 3D map points and 2D pixels, plus intrinsics K, recover [R|t]. The minimal case is P3P — three correspondences give up to four solutions, disambiguated by a fourth point (geometry traceable to Grunert, 1841). EPnP solves n ≥ 4 in O(n). Standard practice: P3P inside RANSAC to find inliers and a rough pose, then Levenberg–Marquardt refinement on all inliers' reprojection error. Phones add a shortcut: the IMU gives gravity, pinning two rotational DoF, so pose search shrinks from 6-DoF to 4, making RANSAC dramatically faster and more robust.

RANSAC — trust arithmetic

Random Sample Consensus (Fischler & Bolles, 1981 — a paper literally written about camera pose estimation): repeatedly fit a model to a minimal random sample, count how many correspondences agree within a pixel threshold, keep the best consensus set. With inlier ratio w and sample size s, the iterations needed for confidence p follow N = log(1−p)/log(1−wˢ) — which is why s matters so much: P3P's s=3 tolerates far dirtier match sets than any larger minimal problem. Modern variants (LO-RANSAC, MAGSAC++, PROSAC) refine locally, marginalize the threshold, and exploit match-quality ordering. The one-line summary an expert carries: matching proposes, RANSAC disposes.

03The two-phase architecture

Every serious VPS — Google's, Niantic's, and the one you could build with open source this weekend — has the same shape: an offline mapping pipeline that turns imagery into a localizable 3D model, and an online localization service that turns one query image into a pose in a few hundred milliseconds.

Why the hierarchy (retrieval first, matching second)? A city-scale map holds hundreds of millions of 3D points; matching a query against all of them is both intractable and self-defeating — with that many candidates, wrong-but-plausible matches swamp RANSAC (perceptual aliasing). Retrieval shrinks the search to a few dozen relevant map images, restoring both speed and precision. Coarse-to-fine is not an optimization; it is what makes city scale possible at all.

04Mapping — structure from motion

Structure from Motion (SfM) takes an unordered pile of images and recovers both the 3D structure (points) and the motion (camera poses) that produced them. The reference implementation is COLMAP (Schönberger & Frahm, 2016) — open source, battle-tested, and the substrate of most academic and many commercial pipelines.

The SfM loop

1. Extract features from every image: keypoints (where) + descriptors (what it looks like). Classically SIFT; in modern pipelines a learned detector like SuperPoint (Section 6).
2. Match pairs of images. Exhaustive for small sets; for large sets, retrieval or GPS priors propose candidate pairs. Verify each pair geometrically with epipolar RANSAC.
3. Initialize from one well-conditioned pair (wide baseline, many inliers): recover relative pose, triangulate initial points.
4. Incrementally register each new image via PnP against the growing point cloud, triangulate new points, and periodically run bundle adjustment.

Bundle adjustment — the gold standard

BA is what makes SfM maps metric-grade: errors are distributed globally instead of accumulating. The Schur complement exploits the structure (points outnumber cameras 1000:1 and don't interact directly) so million-point problems solve in minutes.

Two facts that separate experts from tourists

• Monocular SfM is scale-blind. From images alone, a cathedral and its perfect 10:1 miniature are indistinguishable. Absolute scale must be injected: GPS priors on camera positions, surveyed ground-control points, known-size objects, stereo baselines, or IMU integration. Georegistration — a similarity transform (7-DoF: rotation, translation, scale) aligning the SfM frame to Earth — is what turns a reconstruction into a positioning map with lat/long/altitude output.
• The map is not the point cloud you can see. The localizable map is: 3D points plus their visual descriptors plus the covisibility graph (which points are seen together from where) plus a retrieval index of the mapping images' global descriptors. The pretty dense mesh is a byproduct; localization runs on the sparse, descriptor-bearing skeleton.

Map sources in the wild

05Localization — from one image to a pose

The online path, stage by stage, with the design choices at each:

Stage 0 — the query package

A production query is never just pixels. It carries: calibrated intrinsics (from ARKit/ARCore), gravity direction (IMU) which pins roll/pitch, a coarse position prior (GPS, cell, WiFi — selects which map tile to even consider), and often several frames rather than one. Using priors is not cheating; it is what makes planet scale feasible: GPS narrows the earth to ~100 m, retrieval narrows 100 m to ~10 images, matching narrows 10 images to one pose.

Stage 1 — image retrieval (coarse)

Encode the query into a single global descriptor vector, nearest-neighbor search against the map images' descriptors. The lineage: bag-of-visual-words and VLAD (handcrafted aggregation) → NetVLAD (2016; a CNN with a learnable VLAD pooling layer, trained on Street View time-lapse triplets — the breakthrough that made retrieval robust to day/season change) → AP-GeM, DELG → transformer/foundation-model era: AnyLoc, MixVPR, SALAD built on DINOv2 self-supervised features, which currently dominate place-recognition benchmarks. Retrieval quality gates everything downstream: if the right map images are not in the top-k, no matcher can save the query.

Stage 2 — 2D–3D correspondence (fine)

Extract local features from the query; match them against features of the retrieved map images; each map feature is tied to a triangulated 3D point, so 2D–2D matches lift to 2D–3D correspondences. Cluster retrieved images by covisibility so matches accumulate into one consistent local scene rather than several rival hypotheses.

Stage 3 — pose solving

P3P inside RANSAC (gravity-aided if available) → inlier set → Levenberg–Marquardt refinement of the pose on all inliers. Output: [R|t], inlier count, and reprojection statistics. The inlier count is the confidence signal: below a threshold (commonly a few dozen), a production system reports "no fix" rather than a guess — a wrong pose confidently delivered is far worse than none, because downstream fusion (Section 8) will propagate it.

Failure taxonomy — where queries die

06Features and matching — the forty-year arms race

Handcrafted era

• Harris corners (1988) — where gradients vary in two directions; detection only.
• SIFT (Lowe, 1999/2004) — the field's foundation stone: scale-space extrema detection + 128-D gradient-histogram descriptor, invariant to scale/rotation and robust to moderate viewpoint and lighting change. Still a respectable baseline for mapping; COLMAP's default.
• SURF (2006) — faster SIFT approximation. ORB (2011) — binary, real-time, powers ORB-SLAM; weak under strong appearance change.

Learned era

• SuperPoint (DeTone et al., 2018) — self-supervised CNN emitting keypoints + 256-D descriptors in one pass; trained via "homographic adaptation" (warp images, demand consistency). The workhorse detector of modern VPS.
• D2-Net, R2D2, DISK, ALIKED, XFeat (2019–2024) — variations on detect-and-describe: describe-then-detect, reliability-aware training, differentiable keypoints, lighter/faster backbones for edge deployment.

Matching: from arithmetic to attention

• Nearest neighbor + Lowe ratio test — accept a match only if the best descriptor distance is <~0.8× the second best. Simple, still everywhere.
• SuperGlue (Sarlin et al., 2020) — a graph neural network with self- and cross-attention over both images' keypoints, solving matching as partial assignment (Sinkhorn). Matches with context — a keypoint's neighbors help decide its correspondence — nearly doubling hard-condition localization recall when it appeared.
• LightGlue (Lindenberger et al., 2023) — SuperGlue accuracy at a fraction of the cost via early-exit on easy pairs and pruning of unmatchable points. The current default pairing: SuperPoint + LightGlue.
• Dense/detector-free: LoFTR (2021), RoMa (2024) — match at feature-map level with transformers, no keypoints at all; strongest on texture-poor scenes, at higher compute cost.
• Foundation 3D era: DUSt3R / MASt3R (2024) — transformers that regress 3D pointmaps (and matches) directly from image pairs without calibration, collapsing parts of the classic SfM pipeline into a single forward pass; MASt3R-SfM and successors are re-plumbing mapping itself.

07Alternative paradigms — and why the classic pipeline still wins

Absolute pose regression (APR)

PoseNet (Kendall et al., 2015): train a CNN to map image → 6-DoF pose directly. Elegant, fast, and — as Sattler et al. (2019) demonstrated — fundamentally limited: APR behaves like sophisticated image retrieval, interpolating among training poses rather than reasoning geometrically, so it fails to extrapolate to novel viewpoints and trails structure-based methods by an order of magnitude of accuracy. Verdict: historically important, production-irrelevant.

Scene coordinate regression (SCR)

Instead of regressing the pose, regress each pixel's 3D world coordinate, then solve PnP+RANSAC on the predictions — keeping the geometric engine but replacing the explicit descriptor map with a network that has memorized the scene. Line: SCoRe Forests (2013) → DSAC* (differentiable RANSAC, Brachmann & Rother) → ACE (2023): mapping a new scene in ~5 minutes with a 4 MB network, no depth needed, state-of-the-art in small-to-mid scenes → ACE-G (2025): query pre-training for generalization. The industrial home of this line is Niantic Spatial's research group (nianticspatial.com/research), which is betting on SCR and its descendants as the scalable map format; the same group's map-free relocalization (2022) goes further still — metric pose relative to a single reference photo, no map at all, the natural fit for pop-up AR at places nobody pre-scanned. Weaknesses: scale (city-level SCR remains open) and the retrain-on-change problem. For room-to-building scale with a fixed scene, SCR is the modern dark horse — an expert should know it well.

Implicit/neural maps

NeRF-based localization (iNeRF: invert a radiance field by gradient descent on photometric error) and 3D Gaussian Splatting-based relocalization replace the point cloud with a differentiable renderer: localize by "render-and-compare." Attractive properties: photorealistic maps double as content, analysis-by-synthesis handles viewpoint gracefully. Current status: compelling research, not yet at production robustness/scale; splatting's speed is rapidly closing the practicality gap, and hybrid schemes (feature-based init + render-and-compare refinement) are the likely landing zone.

08VPS + VIO — how production AR actually works

No shipping system calls the VPS every frame. The division of labor:

• On-device VIO (visual-inertial odometry — ARKit/ARCore's core): fuses camera tracking with IMU integration to deliver smooth 6-DoF motion at 30–60 Hz with low latency. Its flaw: the origin is arbitrary and the trajectory drifts (roughly ~1% of distance traveled, worse in feature-poor spaces), and it knows nothing of the world frame.
• VPS fixes — occasional (on session start, on demand, every few seconds-to-minutes, or when drift covariance grows): each fix is an absolute pose measurement in the world frame.
• Fusion — an extended Kalman filter or sliding-window pose-graph estimates the transform world ← odometry, treating VPS fixes as measurements with covariance and outlier gating (a bad fix must be rejectable — this is why honest confidence reporting in Section 5 matters). Corrections are applied gently (blended over frames) so virtual content never visibly teleports.

This architecture is why "VPS latency" of 200 ms is acceptable: the user experiences VIO's latency (~ms), while VPS merely re-anchors the frame occasionally. It is also the template for robotics: swap ARKit for a LiDAR-inertial odometry stack, keep the same global-fix logic.

Anchors — the content layer

An anchor is a named pose in the world frame to which content is attached ("the dragon stands at this plaza corner, facing the fountain"). Persistence (content survives sessions) and sharing (two users see the same dragon in the same spot) both reduce to: everyone localizes into the same map frame, anchors live in that frame. Cloud anchors, geospatial anchors, location anchors — every vendor's flavor is this one idea.

09Production systems survey

10The hard problems

1 · Long-term appearance change

Day↔night, summer↔winter, dry↔snow, renovation. The map was photographed under one condition; the query arrives under another. This is the canonical research problem — the Aachen Day-Night, RobotCar Seasons, and CMU Seasons benchmarks (Section 11) exist precisely to measure it. Mitigations: condition-robust learned features and retrieval (the main line of progress), mapping multiple conditions, semantic filtering (match on buildings, ignore vegetation — geometry stable, appearance volatile), and appearance-transfer augmentation at training time.

2 · Dynamic and ambiguous scenes

Crowds, vehicles, and merchandise occlude and pollute; identical corridors, glass, and chain-store facades alias. Mitigations: semantic masking of transient classes at both mapping and query time, multi-frame queries, position priors, and RANSAC's inherent outlier tolerance. Glass and mirrors remain genuinely unsolved for feature-based methods.

3 · Scale and freshness economics

Mapping the world once is capex; keeping it fresh is opex forever. The engineering answers: change detection (localization failure rates per tile are themselves the staleness signal), incremental map updates (register new imagery into the existing model rather than rebuilding), and crowdsourcing the refresh loop (Section 9's strategic read). Map compression (Section 4) governs serving cost.

4 · Privacy — both directions

• Map privacy: SfM point clouds with descriptors are not anonymous geometry — Pittaluga et al. (2019) showed recognizable images can be reconstructed by inverting descriptors. Proposed defenses: lifting points to lines (Speciale et al., 2019) — later shown partially attackable (Chelani et al., 2021) — descriptor-free geometric matching (GoMatch line), and splitting sensitive computation between device and server. An active cat-and-mouse literature.
• Query privacy: every localization uploads what the user's camera sees, tied to a precise pose. Production mitigations: on-device feature extraction (send descriptors, not pixels), ephemeral processing commitments, and on-device localization against downloaded map tiles (the strongest answer, adopted for sensitive venues).

5 · Integrity and spoofing

Rarely discussed, increasingly relevant as robots act on VPS output: a printed poster of a known facade is the visual analog of GPS spoofing. Defenses mirror GNSS thinking — multi-frame consistency with ego-motion, cross-checking against VIO and coarse radio position, and geometric liveness (a flat poster fails parallax checks across two frames).

11Benchmarks, metrics, and how to read a leaderboard

The metric

Visual localization is scored as recall at pose-error thresholds: the percentage of query images localized within both a position and an orientation bound. The community-standard triple (from the Long-Term Visual Localization benchmark, visuallocalization.net):

Read all three together: the first tells you AR-grade precision, the last tells you gross failure rate. A method with great (0.25 m) recall but poor (5 m) recall fails catastrophically when it fails — often worse in production than a method that degrades gracefully.

The benchmark canon

Reading a leaderboard like an expert

• Check which stage differs between entries — many "new SOTA" rows are the same hloc skeleton with one component swapped. Attribute the gain to the component, not the brand.
• Check compute honesty: dense matchers (RoMa-class) buy recall with 10–100× matching cost; a production system cares about recall per millisecond per dollar.
• Check the night column first on Aachen — day columns have been near-saturated for years; hard conditions are where methods actually separate.
• Distrust single-benchmark wins; the canonical methods (SuperPoint+LightGlue+hloc) win by being uniformly strong everywhere, which is what deployment requires.

12Build your own VPS — the practical recipe

A venue-scale VPS (shop, mall, campus, factory) is a weekend-to-week project on open source. The reference stack:

The recipe

# 1 · CAPTURE — walk the venue; video → frames (1–2 fps), or 200–2000 photos.
#     Overlap generously; revisit from opposing directions; capture at the
#     times-of-day you expect queries. Loop closures matter more than count.

# 2 · MAP — hloc: extract features, match pairs, run COLMAP SfM
extract_features (superpoint)  →  pairs_from_retrieval (netvlad, top-40)
→  match_features (lightglue)  →  reconstruction (COLMAP)
# Output: sparse model (cameras.bin, images.bin, points3D.bin) + feature DB

# 3 · GEOREFERENCE — align the model to your venue frame:
#     mark 3+ surveyed control points in images, solve the 7-DoF similarity
#     (COLMAP model_aligner), or accept the arbitrary frame for AR-only use.

# 4 · LOCALIZE — per query image:
retrieval top-k  →  match query↔retrieved  →  lift to 2D–3D
→  pycolmap absolute_pose_estimation (P3P+RANSAC+refine)
# Returns: pose, num_inliers  →  accept if inliers ≥ ~50 (tune on your venue)

# 5 · SERVE — wrap step 4 in a service; client sends frame + intrinsics
#     (and gravity if available), receives pose + confidence.

The latency budget (single GPU server, 1080p query)

Judgment calls that decide success

• Capture discipline beats algorithm choice. A mediocre pipeline on thorough, multi-condition capture outperforms SOTA on a thin single-pass scan. Budget 70% of effort here.
• Set the acceptance threshold empirically. Walk the venue with ground truth (or self-consistency checks: sequential queries must yield VIO-consistent poses) and choose the inlier cutoff that kills every wrong fix. False fixes are the product-killer.
• Plan the refresh loop on day one. Decide what triggers re-mapping (failure-rate per zone), and make map updates incremental (register new images into the existing model — hloc/COLMAP support this) rather than rebuild-the-world.
• On-device vs server: venue maps of a few hundred MB can ship to the device (privacy, latency, offline); city maps cannot. The hybrid — device extracts features, server matches — is the common production compromise.

13Applications and the business lens

• AR navigation & persistent content — airport/mall/hospital wayfinding with arrows painted on the floor through the camera; city-scale AR layers (Google Maps Live View is VPS-powered). The flagship consumer application.
• Robotics last-meter — warehouse AMRs, delivery robots, and service robots use VPS-style relocalization to recover pose after kidnap/restart and to anchor to semantic infrastructure ("dock #7"), where GPS is absent and wheel+LiDAR odometry drifts.
• Drones in GPS-denied/degraded settings — indoor inspection, under-bridge survey, urban canyons; and as GPS-jamming resilience: terrain/landmark-relative visual navigation is the civilian cousin of military visual-aided inertial navigation.
• Industrial digital twins — technician points a tablet at a pump; the twin overlays torque specs and maintenance history. VPS is the registration layer between the twin and reality (Immersal/Hexagon's core market).
• Automotive lane-level localization — camera-relocalization against crowdsourced sparse maps (Mobileye REM) delivers lane-keeping-grade position where GNSS cannot.
• Accessibility — precise position + heading enables turn-by-turn guidance for blind and low-vision users indoors, a domain where meter-level error is the difference between usable and useless.

The economics in three sentences

Mapping is a fixed cost per venue/city; localization is a near-zero marginal cost per query — classic infrastructure economics with winner-take-most dynamics per geography. The moat is never the algorithm (open source is within a few percent of any proprietary stack); it is the imagery corpus and its refresh loop. Consequently the viable entries are: own a fleet (Google/Apple), own a crowd (Niantic/Mobileye), own the venue relationship (Immersal-style enterprise), or serve where giants won't — private facilities that will never let a fleet inside and want on-premises maps for confidentiality.

14Expert self-test

Fifteen questions. An expert answers ~13 without looking. Answers fold open.

15Glossary

6-DoF pose

Position (x,y,z) + orientation (roll,pitch,yaw); mathematically [R|t], R ∈ SO(3).

Anchor

A named pose in the map/world frame to which virtual content or semantics attach.

Bundle adjustment (BA)

Joint nonlinear least-squares refinement of all poses and 3D points minimizing total reprojection error.

Covisibility graph

Graph recording which map points are observed together from which images; used for clustering and consistency.

Descriptor

A vector summarizing local appearance around a keypoint (local, e.g. SuperPoint 256-D) or of a whole image (global, e.g. NetVLAD) for matching/retrieval.

Epipolar geometry

The two-view constraint (essential/fundamental matrix) relating corresponding points; the verification tool of pairwise matching.

Extrinsics / intrinsics

Pose [R|t] (world→camera) / internal projection parameters K plus lens distortion.

Georegistration

7-DoF similarity alignment (R, t, scale) of a reconstruction to an Earth-fixed frame.

hloc

Hierarchical Localization — the retrieval → match → PnP reference architecture and its open-source implementation.

Inlier

A correspondence consistent with the estimated model within threshold; inlier count ≈ pose confidence.

Keyframe

A selected representative frame retained in a map or SLAM graph.

Perceptual aliasing

Distinct places that look alike (corridors, chain stores), inducing confident wrong retrieval/matches.

PnP / P3P

Perspective-n-Point: pose from n 2D–3D correspondences with known intrinsics; P3P is the 3-point minimal case.

RANSAC

Random Sample Consensus — robust estimation by minimal-sample hypothesize-and-verify voting.

Relocalization

Recovering pose in an existing map after tracking loss or at session start — the VPS operation, viewed from SLAM.

Reprojection error

Pixel distance between a projected 3D point and its observed feature; the universal objective.

Retrieval (VPR)

Visual place recognition: finding which map images/places match a query, via global descriptors.

Scene coordinate regression

Regressing per-pixel 3D world coordinates with a scene-specific network, then PnP (DSAC*, ACE).

SfM

Structure from Motion — joint recovery of 3D structure and camera poses from image collections (COLMAP).

SLAM / VIO

Simultaneous Localization and Mapping / Visual-Inertial Odometry — online tracking (+mapping) in unknown space; VIO is the camera+IMU odometry core of ARKit/ARCore.

Triangulation

Intersecting rays from ≥2 posed views to place a 3D point.

WiFi CSI

Channel State Information — per-subcarrier amplitude/phase of the WiFi channel, a multipath fingerprint of a space; enables device-free sensing (presence, gesture, fall, respiration) and sub-meter radio positioning (IEEE 802.11bf).

16Reading list — the canon, in reading order

1. 1981Fischler & Bolles — RANSAC ("Random Sample Consensus") — robust estimation, born from camera pose.
2. 2004Lowe — SIFT ("Distinctive Image Features from Scale-Invariant Keypoints").
3. 2004Hartley & Zisserman — Multiple View Geometry in Computer Vision — the field's textbook; chapters on projection, two-view geometry, and BA are the mathematical bedrock.
4. 2009Lepetit et al. — EPnP — O(n) PnP.
5. 2015Kendall et al. — PoseNet — absolute pose regression (read with its 2019 refutation, below).
6. 2016Schönberger & Frahm — COLMAP ("Structure-from-Motion Revisited").
7. 2016Arandjelović et al. — NetVLAD — learned place recognition.
8. 2018DeTone et al. — SuperPoint — self-supervised keypoints.
9. 2018Sattler et al. — Benchmarking 6DOF Outdoor Visual Localization in Changing Conditions — Aachen/RobotCar/CMU benchmark paper.
10. 2019Sarlin et al. — From Coarse to Fine: Robust Hierarchical Localization at Large Scale — the hloc paper; the modern VPS blueprint.
11. 2019Sattler et al. — Understanding the Limitations of CNN-based Absolute Camera Pose Regression — why APR ≈ retrieval.
12. 2019Pittaluga et al. — Revealing Scenes by Inverting Structure from Motion Reconstructions — the privacy attack.
13. 2020Sarlin et al. — SuperGlue — attention-based matching.
14. 2021Sun et al. — LoFTR — detector-free transformer matching.
15. 2023Lindenberger et al. — LightGlue — fast adaptive matching.
16. 2022Arnold et al. — Map-free Visual Relocalization — metric pose from a single reference image, no map at all.
17. 2023Brachmann et al. — ACE ("Accelerated Coordinate Encoding") — 5-minute scene coordinate mapping.
18. 2023Keetha et al. — AnyLoc — foundation-model (DINOv2) place recognition.
19. 2024Wang et al. / Leroy et al. — DUSt3R / MASt3R — pointmap regression; the foundation-model turn in 3D matching and SfM.

Live resources: visuallocalization.net (benchmark + leaderboard) · github.com/cvg/Hierarchical-Localization (hloc) · colmap.github.io (COLMAP docs) · Image Matching Challenge (CVPR, annual) for the state of matching. Watching: YouTube @bilawalsidhu — Bilawal Sidhu, formerly the Google product lead behind 3D Maps / ARCore Geospatial (the Google VPS of Section 9), now a 2M-subscriber explainer of reality capture, Gaussian splatting, and spatial intelligence; the moving-picture companion to this report.

17Plain speak — the whole report, retold simply

1 · What it is

A VPS tells a camera exactly where it is and which way it is pointing, by recognizing what it sees against a prepared 3D memory of the place. GPS guesses your dot from satellites — meters off, no idea which way you face, dead indoors. A VPS is accurate to centimeters, knows your facing, and works indoors — but only where someone has already built the map. Hold one idea above all: it is not a sensor, it is recognition against a map. One cousin worth knowing: WiFi CSI — the router's radio echo of a room. It can sense people moving even through walls, but it can never deliver the precise position-plus-facing a camera can; the two work best together.

2 · The geometry

A camera is a machine that drops 3D points onto a flat sensor. If you know which 3D points landed on which pixels, there is only one place the camera can be standing — and finding that place is the whole game. "Reprojection error" is simply how far off the landing predictions are for a guessed position; every algorithm in the field works to make that number small. Three good point-to-pixel pairs already give a candidate answer. And because many pairs are wrong, the answer is chosen by a voting trick: try many small random samples, keep the answer the most pairs agree with. Matching proposes; voting disposes.

3 · The two phases

Build the map once, slowly and expensively: photograph everything, compute a 3D model. Then answer questions forever, fast and cheaply: recognize, match, solve. It is the same shape as a search engine — crawl offline, serve online — and just like search, whoever owns the index owns the business.

4 · Making the map

Take many overlapping photos. Software finds the same small details across different photos and works out, simultaneously, where every photo was taken and where every detail sits in 3D. A final clean-up pass spreads the small errors evenly so nothing accumulates. One catch: photos alone cannot tell true size — a dollhouse and a house photograph identically — so real-world scale must come from outside: GPS, measured reference points, or the phone's motion sensors. The finished map is a cloud of 3D dots, each remembering what it looks like. That memory is what queries match against.

5 · Answering a query

Three shrinking questions. Which part of town? — a fast whole-image similarity search picks a handful of map photos. Which details match? — compare the small features between the query and those photos. Where must the camera stand so that everything lines up? — geometry gives the spot and the facing. Then the honesty meter: count how many matches truly agree with the answer. If too few agree, say "no fix." A confident wrong answer is the worst thing this system can produce, because everything downstream trusts it.

6 · Recognizing details

A "feature" is a small recognizable detail — a corner, an edge, a mark. For twenty years, hand-written math decided what counts as a detail and whether two look alike. Today, neural networks learn both — and they read the whole image for context before deciding any single match. Each generation of this technology survives worse conditions than the last: night, rain, snow, and big changes of viewpoint.

7 · The rival ideas

Three challengers have tried to replace the classic recipe. Let a network look at a photo and guess the position outright — it turned out to memorize rather than measure, and fails away from its training shots. Let a network memorize one scene's 3D shape so every pixel can name its 3D point — this genuinely works for a single building, but must be retrained when the scene changes and does not scale to cities. Make the map photorealistic and search inside it like a rendering — promising and young. At city scale, the classic recipe still beats all three.

8 · Living with the phone's own tracking

Phones already track their own motion smoothly, using the camera and motion sensors together — but like a step-counter, they slowly wander off course. The VPS plays the role of the trail sign: every so often, look up, check, correct. The user experiences only the smooth track; corrections are blended in so nothing visibly jumps. "Anchors" are named spots saved in the shared map where content lives — which is why two people's phones can show the same virtual object standing on the same real corner.

9 · Who runs the big ones

Google built its VPS from Street View — it covers much of the world's streets. Niantic built its map from millions of players' phone scans — strongest at interesting places. Apple flew and drove its own fleet — select cities, tightly integrated. Immersal walks your building for you — venues and factories. Microsoft ran an anchor service with no camera corpus of its own, and it is the one that shut down. The lesson in one line: the map is the moat, not the math.

10 · What is still hard

The map is a photograph of the past; every query is the present. Night, snow, and renovations break the resemblance. Identical corridors and repeated storefronts fool the recognizer. Keeping maps fresh costs money forever. Privacy cuts both ways: map data can be reverse-engineered back into images of the place, and every query shows a server what your camera sees. And the system can in principle be spoofed with a printed picture of a place — though a poster is flat, and flat things fail the parallax check when the camera moves.

11 · Keeping score

The standard exam: out of 100 query photos, how many were placed within 25 centimeters and 2 degrees of the truth? Also watch the loose 5-meter score — when the system misses, it should miss near, not teleport across town. And always read the night-time results first; the daytime exam was solved years ago.

12 · Build one yourself

One person, one phone, a weekend, free software. Film the venue walking slowly, from both directions, at the times of day you care about. Free tools turn the footage into a 3D map. A small program answers queries: similarity search, detail matching, geometry. Spend your effort where it counts — thorough filming beats clever algorithms — and set the "no fix" threshold strictly, so a wrong answer never escapes. Decide on day one how you will re-map when the venue changes, because it will.

13 · Where the money is

Anywhere someone or something needs "exactly where am I, facing which way" and GPS cannot answer: navigation arrows painted through the camera in malls, airports, and hospitals; robots finding their dock; drones where GPS is absent or jammed; a factory tablet that recognizes the machine it is pointed at; cars holding their lane. The cost model: expensive to map once, nearly free per query after — value concentrated exactly where GPS is blind.

14–16 · The wrap-up, plain

Test yourself with five one-breath questions: why does a VPS know your facing while GPS never can; why can photos alone not tell true size; why does matching against the whole map at once fail; why is "no fix" better than a wrong fix; why did the service without its own cameras die? Answer those cleanly and Section 14's full exam will go well. The vocabulary reduces to five items: pose = position + facing; features = recognizable details; structure-from-motion = photos in, 3D out; PnP + RANSAC = geometry + voting; hloc = the standard recipe wiring it all together. And if you read only four papers, read them in this order: SIFT, COLMAP, hloc, SuperGlue — the detail, the map, the recipe, the matcher.