The Robot Sensor Stack: How Cameras, Lidar, and Touch Become One State Estimate

On a workbench, every sensor lies to you a little. A camera hands back pixels with no notion of distance. A lidar returns a cloud of ranges with no notion of what it hit. A wrist sensor emits six numbers, an encoder emits a joint angle, and an IMU emits an acceleration signal that drifts within seconds. None of these is a decision, and none is trustworthy alone. The robot sensor stack exists to reconcile them, and the quiet engineering that turns those mismatched streams into one usable picture is where most real-world robotics actually happens — the perception half of what our field guide to physical AI calls the sense-plan-act loop.

What is the robot sensor stack?

The robot sensor stack is the layered set of sensors a robot uses to perceive two things at once: the outside world, and its own body. A typical stack spans RGB and depth cameras, lidar for ranging, six-axis force-torque sensors at the wrist, joint encoders and an inertial measurement unit (IMU) for proprioception, and, on manipulators that grasp, tactile sensors at the fingertips. It is the foundation of robot perception. Each modality covers a blind spot of the others: cameras give semantics but weak geometry, lidar gives geometry but no color or contact, and force and tactile sensors give contact but see nothing beyond the point of touch. This piece is part of our Physical & Embodied AI Foundations cluster.

So how does a robot turn raw signals into something a policy can act on? In three stages. First, each modality is preprocessed and time-synchronized: undistorted, rectified, normalized, and timestamped to a common clock. Second, the streams are fused, either into an explicit low-dimensional state estimate by a filter such as an Extended Kalman Filter (EKF), or implicitly inside a learned policy that ingests them as input tokens. Third, that fused state is handed to a controller or a vision-language-action (VLA) model, which outputs motion. Perception is the first two stages; the glossary covers the terms.

How does the sensor-to-policy pipeline actually run?

In practice the pipeline runs at several clocks at once. Fast, cheap signals (the IMU at hundreds to thousands of hertz, joint encoders with sub-microsecond latency) drive the high-rate prediction of where the robot is. Slower, richer signals (camera frames, lidar sweeps) arrive less often and correct that prediction. On a legged or humanoid robot, a mid-level error-state EKF commonly runs at around 1 kHz, fusing the body IMU, limb IMUs, forward kinematics, and contact detection into a floating-base state estimate. Much of this plumbing lives in middleware such as ROS 2 (the current release, “Kilted Kaiju,” reached general availability in May 2025); the framework does not do the fusion for you, but it standardizes how timestamped sensor messages move between nodes.

Cameras: what do RGB, depth, and stereo each buy you?

An RGB camera is the cheapest, densest source of semantic information a robot has: texture, color, object identity, text. What it does not give you natively is depth, and it needs light. Robots recover depth in three main ways, each with a different failure mode.

Passive stereo uses two RGB cameras and computes disparity between them, increasingly with a neural network, the way two eyes do. It works indoors and outdoors, reaches useful range to roughly 20 m, and needs no emitter, but its accuracy degrades with distance, from around 1% of range up close to as much as 9% at the far end. It struggles on textureless surfaces and wants a GPU. The Stereolabs ZED series is a common example.

Time-of-flight (ToF) emits modulated infrared light and measures the phase shift of the return. It produces dense depth at low compute over a short band, roughly 0.2 to 5 m, but suffers multi-path interference in corners and concave geometry, where light bounces before it comes back.

Structured light projects a known infrared pattern and reads depth from how the pattern distorts. It is accurate at short range (the Orbbec Astra series is a familiar example) but washes out in bright ambient or outdoor infrared.

One structural note for anyone speccing hardware in 2026: Intel wound down parts of its RealSense line (lidar, face authentication, tracking) between roughly 2021 and 2023, then spun the stereo-depth business out as an independent company in mid-2025, backed by a reported $50 million from Intel Capital and the MediaTek Innovation Fund. RealSense cameras remain widely deployed, but the vendor landscape shifted.

Lidar: when do you actually need it?

Lidar measures range directly by timing laser pulses, producing precise 3D geometry that neither cameras nor contact sensors can match, which is why it anchors most outdoor navigation and SLAM. The trade-offs are cost, mechanical complexity in spinning units, sparse vertical resolution, and degradation in rain, fog, and dust.

The market is growing but the numbers are soft. One market-research estimate (MarketsandMarkets) puts the solid-state lidar market at roughly $0.50 billion in 2025 rising to about $3.20 billion by 2030; automotive lidar figures from the same source family run from around $1.70 billion in 2026 to about $11.26 billion by 2033. Treat these as estimates, because projections vary widely between research firms. On the supply side, Hesai (China) is widely reported as the volume leader, surpassing 2 million cumulative lidar deliveries in 2025 and expanding capacity toward 4-plus million units in 2026; RoboSense, Ouster, Velodyne, Luminar, and Innoviz are the other names that recur across reports. For an indoor manipulator, lidar is often overkill; for a mobile robot outdoors, it is frequently the backbone.

Force-torque sensors: how does a robot feel what it pushes?

A six-axis force-torque (F/T) sensor measures three forces (Fx, Fy, Fz) and three torques (Tx, Ty, Tz) at once, usually mounted at the wrist between the arm and the tool. This is what lets a robot insert a peg, wipe a surface, or stop pushing before it breaks something. ATI Industrial Automation is a commonly cited maker, and its range illustrates the design space: the Axia80 streams over Ethernet at sample rates up to 8 kHz, compact capacitive models sit around 200 N and 4 Nm, the Nano17 (17 mm across) is among the smallest six-axis sensors made, and the Omega331 handles very large loads (up to roughly 21,000 lbf in Fz). Those are one vendor’s specs, not an industry standard.

Force sensing also sits at the center of collaborative-robot safety. The 2025 revision of ISO 10218, the core industrial-robot safety standard republished in February 2025, now folds the biomechanical force and pressure limits from ISO/TS 15066 into its companion volume, ISO 10218-2:2025 (the applications and system-integration part), making “how hard may a robot touch a human” a first-class requirement rather than a separate technical specification.

Proprioception: how does a robot know where its own body is?

Proprioception is the robot’s sense of its own configuration, and it is the most reliable data in the whole stack because it is generated internally. Joint position typically comes from two encoder types working together. Incremental encoders report change in position at very low latency and high resolution (10,000-plus counts per revolution, sub-microsecond latency), but they only measure relative motion, so they need a homing move to establish a zero. Absolute encoders, usually magnetic or optical, report true position at power-on with no homing, commonly at around 14-bit resolution in modern designs. High-end arms increasingly run both, over digital interfaces such as BiSS-C or ABZ. Add an IMU (high-rate linear acceleration and angular velocity) and you have the fast backbone that a fusion filter propagates between slower camera and lidar corrections. The catch: an IMU drifts quickly on its own, which is exactly why it must be fused, never trusted raw.

Tactile sensing: how does a robot feel contact at the fingertip?

Tactile sensing is the newest layer and the one closest to a research frontier. The dominant modern approach is optical (vision-based) tactile sensing: a small camera inside the fingertip images a deformable gel or elastomer skin, and contact geometry, force distribution, and slip are inferred from how the skin deforms. GelSight pioneered the technique; Meta AI’s open-hardware DIGIT sensor made it cheap and reproducible, outputting 320x240 images at 60 fps in a fingertip smaller than a GelSight Mini. A more advanced joint GelSight-Meta design, Digit 360, reportedly combines 18-plus sensing features for micron-level touch (a claim that leans on the vendor’s own materials, so corroborate it cautiously). GelSight was also reported to have won a Phase II U.S. Air Force SBIR contract in March 2026 for a ruggedized tactile fingertip; that is single-source as of this writing, so treat it as unconfirmed. The trade-off with optical tactile is blunt: extraordinary local detail across a tiny field of view, plus a camera and its compute crammed into every fingertip, and a soft skin that wears.

The sensor stack at a glance

Modality	What it measures	Strength	Where it fails
RGB camera	Color, texture, semantics	Cheap, dense, rich context	No native depth; needs light
Stereo depth	Depth via disparity	Passive, indoor/outdoor, ~20 m	Error grows with range (~1% to ~9%); needs texture + GPU
ToF depth	Depth via IR phase shift	Dense, low compute (~0.2-5 m)	Multi-path errors in corners/concavities
Structured light	Depth via projected IR pattern	Accurate at short range	Washes out in bright/outdoor IR
Lidar	Range via laser time-of-flight	Precise long-range 3D geometry	Cost; sparse verticals; rain/fog/dust
Force-torque (6-axis)	Fx/Fy/Fz + Tx/Ty/Tz at wrist	Direct contact force, up to ~8 kHz	Senses one mount point only; needs calibration
Joint encoders	Joint angle/position	Very low latency, high resolution, internal	Says nothing about the world
IMU	Linear acceleration + angular rate	Very high rate	Drifts fast; useless unfused
Optical tactile	Fingertip contact geometry, force, slip	Micron-scale local detail	Tiny field of view; wear; per-finger compute

How does sensor fusion combine everything into one state estimate?

Sensor fusion is the step that turns many partial, noisy views into one estimate the rest of the robot can trust. The classical workhorse is the Kalman filter family: the Extended Kalman Filter (EKF) and Unscented Kalman Filter (UKF). The pattern is consistent across mobile and legged robots. The high-rate IMU drives a prediction step, and slower measurements (wheel or leg odometry, lidar, visual-inertial camera data, GNSS outdoors) arrive as correction steps, collapsing everything into a single pose or state vector. More recent field-robotics work has shifted toward optimization-based fusion (factor graphs and bundle-adjustment-style methods), which trade the filter’s constant-time update for higher global accuracy and more flexibility. That is a documented trend, not a single event.

Classical filters or learned fusion: which paradigm wins?

There is now a second paradigm. In modern robot-learning pipelines, fusion increasingly happens implicitly inside a learned policy. Rather than an engineer hand-designing an EKF that emits a tidy state vector, a transformer-based vision-language-action model takes raw or lightly processed modalities (RGB, depth, proprioception, sometimes tactile) as input tokens and learns its own internal representation. This is the lineage from Google DeepMind’s RT-2 (July 2023), the first major VLA, through the open-source OpenVLA (a 7-billion-parameter model trained on 970k real demonstration episodes; released around mid-2024, though some sources date it later), to NVIDIA’s Isaac GR00T N1 (announced March 2025), a humanoid foundation model with an explicit fast “System 1” and slow “System 2” split. NVIDIA has since previewed a GR00T N2 with claimed large gains, but as of mid-2026 that is forward-looking; treat the leaderboard and success-rate claims as unverified until it ships.

Which paradigm wins depends on the task, and the honest answer is that they coexist. Classical filter-based fusion still dominates navigation and localization (SLAM), where an explicit, debuggable pose estimate matters and the physics is well understood. Learned fusion is ascendant in manipulation, where the mapping from messy multi-modal contact to action is hard to write by hand. And a caution worth keeping: vendors talk freely about “end-to-end” learning, but exactly how much of any shipped commercial product replaces a classical filter is largely undisclosed. Assume most real systems are hybrids until told otherwise.

Field notes: what this means when you build

The practical takeaway is unglamorous. A robot’s competence is rarely limited by any single sensor; it is limited by time synchronization, calibration, and honest handling of each modality’s failure mode. Add tactile only if you touch things, skip lidar indoors, never trust an IMU raw, and remember that a fused state estimate is a hypothesis, not ground truth. Whether the fusion lives in a Kalman filter or the weights of a VLA, the job is the same: give the policy the least-wrong picture of the world and the robot’s own body that the hardware allows. For where sensing sits relative to compute, policy, and safety, see our full breakdown of the physical AI stack.