New foundational model StereoAnything aims to enhance stereo matching in computer vision

Recent advancements in artificial intelligence, especially within the realm of computer vision, are paving new paths for various applications across industries. Central to these developments is the application of foundation models that enhance performance in areas such as object recognition, image segmentation, and depth estimation. A significant focus of ongoing research is stereo matching, which is pivotal for the operation of technologies in sectors such as robotics, autonomous vehicles, and augmented reality.

Stereo matching enables the perception of depth and the creation of three-dimensional views of scenes, but researchers face challenges in leveraging foundation models effectively due to the difficulty in obtaining reliable disparity ground truth data. Although numerous stereo datasets exist, integrating these datasets into training models has proven complex and often insufficient for developing ideal foundation models.

Current research is shifting focus to the Stereo-from-mono approach, which seeks to generate stereo-image pairs and disparity maps from single images. Despite this promising direction, the method has only produced approximately 500,000 data samples—an amount deemed inadequate for training robust models capable of effective generalisation under diverse real-world conditions. Earlier stereo-matching methods predominantly employed hand-crafted features, evolving into convolutional neural network (CNN) based models such as GCNet and PSMNet which utilised 3D cost aggregation to improve accuracy. However, video stereo matching, which incorporates temporal data to enhance consistency, still struggles with generalisation.

To address these limitations, a collaborative research effort led by institutions such as Wuhan University’s School of Computer Science, and the Institute of Artificial Intelligence and Robotics at Xi’an Jiaotong University, among others, resulted in the creation of a new foundational model named StereoAnything. This model is designed to deliver high-quality disparity estimates for any matching stereo images, regardless of the complexity of the scene or environmental conditions. StereoAnything’s architecture comprises four main components: feature extraction, cost construction, cost aggregation, and disparity regression, thereby enhancing its robustness.

The researchers optimised the model’s ability to generalise by utilising supervised stereo data without depth normalisation, critical to stereo matching as it relies heavily on scale information. The training process began with a single dataset before amalgamating several top-ranked datasets to bolster robustness. For the generation of realistic stereo pairs, monocular depth models predicted depth, which were then converted into disparity maps using forward warping techniques that also filled gaps and occlusions by leveraging textures from other images.

Evaluations of the StereoAnything framework employed standards such as the OpenStereo and NMRF-Stereo baselines with Swin Transformer for feature extraction. The experiments used the AdamW optimiser, OneCycleLR scheduling, and engaged in thorough fine-tuning across labelled, mixed, and pseudo-labelled datasets, ensuring robust data augmentation was in place. Testing conducted on datasets such as KITTI, Middlebury, ETH3D, and DrivingStereo indicated that StereoAnything made substantial progress in reducing errors, markedly improving upon previous models. Notably, the NMRF-Stereo-SwinT model evidenced a mean error reduction from 18.11 to 5.01, while further fine-tuning StereoCarla with diverse datasets achieved the best mean metric of 8.52%.

The results from this research illustrate that StereoAnything maintains robust performance across varied domains, spanning both indoor and outdoor environments. The outcomes reflect superior accuracy in disparity mappings compared to the preceding NMRF-Stereo-SwinT model, showcasing superior generalisation capabilities across environments with diverse visual and environmental discrepancies.

Ultimately, the innovative StereoAnything model not only addresses the challenges faced by existing stereo matching techniques but also leverages a new synthetic dataset, StereoCarla, to improve model generalisation in varying conditions. The exploration into the effective utilisation of labelled and pseudo-stereo datasets highlights the critical importance of diverse data sources in enhancing the robustness of stereo models. Such advancements may serve as a foundation for further research and improvements in the field of machine learning and computer vision.

Source: Noah Wire Services