Neuromorphic vision and event-based algorithms for driver monitoring

Driver Monitoring Systems (DMSs) are a critical element of advanced and autonomous vehicles, requiring robust analysis of driver state under diverse conditions. Conventional frame-based cameras (FBCs) face limitations in dynamic range, efficiency, and temporal resolution. Event cameras (ECs) asynchronously report luminance changes with extremely high speeds and low latency. Packaged with a dynamic range exceeding 120~dB and power consumption measured in milliwatts, they make a compelling alternative sensor. This thesis investigates how ECs can be exploited for DMSs, spanning early feasibility studies, more advanced and practical multi-task architectures, and novel processing strategies for event representation in high-speed applications. The work begins with a survey of event cameras in automotive sensing, examining event representations, their current use for in-cabin monitoring tasks, the potential of synthetic events, and the most prominent challenges. Building on this foundation, proof-of-concept studies demonstrate the first event-based detectors for yawning and seatbelt state using a recurrent CNN with self-attention. Yawns, a motion-rich behaviour, were successfully detected using a conventional approach for event accumulation, and state-of-the-art performance was achieved on events simulated from public RGB data. In contrast, the largely static seatbelt task required a tailored accumulation strategy to ensure consistent feature visibility, leading to the first demonstration of seatbelt detection from events. Expanding beyond the detection of individual behaviours, a unified multi-task framework was developed for simultaneous facial landmark and blink detection. By modifying an existing architecture to include a convolutional LSTM and a blink detection head, the system achieved robust landmark localisation and blink detection across five datasets, comprising real and simulated events. The results highlighted a trade-off between event representations: histograms offered temporal sharpness that proved advantageous in blink detection, while time surfaces presented a more consistent facial structure and improved landmark accuracy. This trade-off prompted the final work developed over the course of this PhD: Locally Adaptive Decay Surfaces (LADS), a new family of event representations that modulate temporal decay according to local signal dynamics. By preserving structure in static regions while maintaining precision in areas of rapid change, LADS consistently outperformed conventional histograms and globally decayed time surfaces in face and landmark detection. Notably, LADS sustained high accuracy even at 240~Hz update rates, setting new benchmarks for event-based facial analysis. Together, these contributions establish the feasibility and advantages of event cameras in DMSs, demonstrate their potential for complex multi-task monitoring, and introduce adaptive event representations that exploit their unique sensing characteristics. The findings show how neuromorphic vision can enhance the capability and efficiency of future driver monitoring technologies.

Funder

Taighde Éireann – Research Ireland

Publisher

University of Galway

Rights

CC BY-NC-ND

cbn

Collections

University of Galway Theses (PhD Theses)