On the impact of rain on computer vision in automotive applications

The core image processing and machine learning technologies used to enable Connected and Autonomous Vehicles (CAV) are typically developed in ideal conditions, particularly good weather conditions. Given the functional safety requirements expected of such systems, they must also operate reliably under adverse environmental conditions; however, there is a comparative lack of research into the performance of such technology in adverse weather conditions, compared to benign weather conditions. This thesis addresses this gap by investigating the impact of rain on technology for CAV, with a particular focus on visible-spectrum cameras. The primary contributions of the thesis include: - Firstly, a comprehensive review of the literature, including the effects of rain on the imaging pipeline and its downstream impact on deep learning-based perception algorithms. Inspired by communications system design, a novel “Image Formation Framework” (IFF) that treats image formation and perception as a communications problem, is introduced to segment the problem space, offering insights into the changes caused by rain as perceived by automated vehicles. The literature review also evaluates how well publicly available datasets capture these effects, revealing a significant gap in academic research concerning the impact of rain on autonomous vehicles. - Secondly, a detailed study investigates the performance of state-of-the-art object detection algorithms when applied to rainy versus clear images, using the Berkeley Deep Drive (BDD) dataset of real-world images. In addition to high level object detection results, the study includes an in-depth error analysis, identifying where and how the models fail under adverse conditions. Additionally, an object-level analysis is presented based on pixel-level metrics. The limited meta-data included in publicly available datasets restricted the analysis of factors such as rain intensity, droplet distribution, and scene-dependent visibility degradation. This study underscores the need for custom datasets, or enhanced labelling in existing datasets, to better capture the variability introduced by adverse weather. - To gain a more comprehensive understanding of the effects of rain on automotive cameras under controlled conditions (in contrast to real-world conditions), experimental data were collected at the AVL Mobility and Sensor Test Center in Roding, Germany. This controlled environment enables the collection of data while isolating the effects of rain from other environmental factors, enabling direct comparisons between rain and no-rain conditions and minimising the impact of confounding variables. - Using the data collected in controlled conditions, two key studies were completed, focusing, firstly, on the degradation of image quality due to rain, and secondly, the subsequent effect of rain on perception performance using deep learning models. These studies allowed for deeper insights into the impact of rain on camera images, and object detection systems. - Finally, a computationally-efficient framework for simulation of rain was developed (working with an industry collaborator). Overall, the thesis contributes important insights into the effect of rain on the computer vision pipeline in CAV, that goes beyond high-level object detection metrics to encompass detailed object and image-level analysis of the effects of rain. The contributions should provide useful information to designers of computer vision systems for CAV applications operating in inclement weather conditions.

Funder

Taighde Éireann - Research Ireland
European Regional Development Fund
Valeo Vision Systems

Publisher

University of Galway

Rights

CC BY-NC-ND

cbn

Collections

University of Galway Theses (PhD Theses)