Jayanti Rohit Sreekanth supervised by Prof. K Madhava Krishna received his Master of Science by Research in Computer Science Engineering (CSE). Here’s a summary of his research work on Geometry-Enabled Models for Robust Robot Perception

Imagine an embodied agent navigating a complex, unstructured environment. To operate effectively, whether it is retrieving a specific object or building a comprehensive map of a building, the robot must reliably recognize and associate the same physical entities across different points in time and space. Historically, this capability relied on matching sparse, isolated pixels. However, the paradigm of visual correspondence is shifting toward segment matching: establishing connections between semantically and geometrically coherent regions. By reasoning over entire objects or structural surfaces rather than individual points, robots can achieve a higher-level, interpretable, and fundamentally more robust understanding of their surroundings.

Despite its vast potential, segment matching breaks down under wide-baseline conditions. When an agent revisits a scene from an opposing viewpoint and experiences extreme perspective distortion, scale variation, and camera rotations up to 180◦, the 2D appearance of objects changes drastically. Current state-of-the-art vision foundation models and video propagators, trained primarily on 2D priors, struggle severely in these scenarios. Their routine failure modes include perceptual instance aliasing (confusing identical but distinct objects) and topological collapse (failing to recognize the exact same object from a new angle), ultimately leading to fragmented maps and navigational failures.

To resolve this blindspot, this thesis introduces SegMASt3R (Geometry Grounded Segment Matching), a novel architecture that repurposes the strong spatial inductive biases of a 3D Foundation Model (3DFM) for robust, wide-baseline segment association. The proposed pipeline leverages a frozen MASt3R backbone to extract dense, 3D-aware patch features. A newly designed segment-feature head then aggregates these dense representations into compact, instance-discriminative descriptors. To gracefully handle the reality of occlusions and restricted fields-of-view, the assignment process is formulated through a differentiable optimal transport layer. By employing a Sinkhorn solver equipped with a learnable dustbin parameter, the network learns to explicitly reject unmatchable segments while enforcing strict geometric consistency for valid pairs.

Extensive empirical evaluations demonstrate that SegMASt3R achieves state-of-the-art accuracy in wide-baseline segment association. It outperforms existing 2D foundational baselines and dense local feature matchers by up to 30% on the AUPRC metric across challenging indoor benchmarks, including ScanNet++ and Replica. Furthermore, the learned geometry-grounded representations exhibit strong zero-shot generalization to unconstrained outdoor environments, such as the MapFree dataset. Finally, this thesis validates the practical utility of SegMASt3R in downstream robotic perception pipelines, demonstrating significantly reduced identity fragmentation in 3D instance mapping and enhanced localization success in object-relative topological navigation.

July 2026