Vehicle Perception System Based on Heterogeneous Neural Network Sensor Fusion

Abstract

Previous research conducted in the CMHT lab at McMaster University led to the successful development of a Light detection and ranging (LiDAR)-based vehicle perception system, notable for its highly accurate detection of highway objects and moderate classification capabilities. This thesis builds upon that foundation, enhancing the existing system by incorporating sensor fusion with Infrared (IR) cameras and standard cameras. The implementation of sensor fusion significantly augments the system’s performance, enabling it to detect and classify objects effectively under adverse weather conditions and in poor lighting. Key contributions of this research include: • The development of a hardware-synchronized sensor system, blending LiDAR, IR cameras, and cameras. • The creation of a comprehensive sensor calibration process and a novel multi-sensor dataset, the first of its kind to include annotations under various lighting and weather conditions. • The development of an innovative ground segmentation technique using the Savitzky-Golay filter and peak detection, significantly improve the speed of ground point elimination. • The introduction of a novel optimizer cm-reSVSF for complex-valued neural networks, demonstrating superior performance compared to traditional algorithms like ADAM and SGD.