2026-10-05

TerraSight: terrain perception that cannot say "safe" by accident

TerraSight is onboard perception for autonomous planetary rovers, built for Smart India Hackathon. It turns stereo and RGB camera feeds into a live 3D terrain map, classifies every patch into decision zones, and scores it for construction. I built the mission dashboard: the React and Three.js view that renders the terrain, rover paths, classified zones and mission data.

The problem

A rover has to answer one dangerous question on its own: is this ground safe to build on? A wrong "no" means surveying somewhere else. A wrong "yes" sinks a habitat. And the computer answering it is slow, low on power and short on memory, with no labelled dataset of real planetary imagery to train on.

The approach

Being over-cautious is fine. Being falsely reassuring is not. So perception measures, and one deterministic layer decides. Nothing else in the system is allowed to make a safety call.

Architecture

stereo + RGB frames
   |
   v
perception (can be wrong): segmentation, depth, SLAM
   |  slope, roughness, material class, crater distance, confidence
   v
scoring (deterministic): safety score, then zone by strict precedence
   |
   v
API: /map/tiles  /rover/path  /sites  /boundaries
   |
   v
dashboard (Next.js + Three.js): 3D terrain, zones, rover path, ranked sites

The four zones are construction-safe, navigation-only, geological interest and hazardous. When signals conflict the precedence is hazard, then geological, then navigation, then safe.

What was built

A frozen API contract. Four endpoints were fixed on day one. The frontend was built against mock JSON in those exact shapes, then the mocks were swapped for the live FastAPI endpoints.
A classifier with no neural network. Each pixel becomes a six-number feature vector and is labelled by the nearest class centroid. Confidence comes from the margin between the nearest and second-nearest class, and is capped at 0.6 because a heuristic should not claim more.
INT8 quantization of the centroids, verified with zero class flips across every centroid and test cell.
Deterministic scoring. One file turns measurements into a verdict with no ML and no randomness.
The dashboard, deployed at terrasight-liard.vercel.app.

Engineering decisions

Uncertainty pulls toward caution, in the arithmetic. The material's effect on the score is scaled by confidence, so a low-confidence reading of good soil collapses to a neutral value that cannot earn approval.
Missing data scores zero. A dropped sensor produces NaN, and NaN is never treated as neutral or as flat ground.
Safe needs four independent gates. A score of at least 0.70, enough confidence, low roughness and a buildable material. No single number approves a site.
Hazard is checked first. A crater floor that looks flat and smooth is still hazardous, because the class forces it.
Every stage sits behind a fixed interface, so a trained model can replace the classifier later without a rewrite.

What failed

The textbook approach, a U-Net, was not available: there was neither the labelled data nor the compute to train and run one. The project shipped without a neural network because that was the version that could be checked.

Current limitations

The timing numbers (about 1 ms for segmentation and 5.4 ms for a full run) are from a development laptop. They exist to catch large regressions, not to certify flight hardware.
The pipeline has been run on simulated datasets. Depth-map calibration on more datasets is still to do.
A MobileNetV3-Small model is planned as a drop-in, and it ships only if it beats the classical classifier.