Radar and Camera Fusion in Intelligent Transportation System

Abstract

Modern smart cities often consist of a vast array of all-purpose traffic monitoring systems to understand city status, help reduce traffic congestion, and to enforce traffic laws. It is critical for these systems to be able to robustly and effectively detect and classify road objects. The majority of current traffic monitoring solutions consist of single RGB cameras. While cost-effective, these RGB cameras can fail in adverse weather or under poor lighting conditions. This thesis explores the viability of fusing an mmWave Radar with an RGB camera to increase performance and make the system robust in any operating conditions. This thesis discusses the fusion device's design, build, and sensor selection process. Next, this thesis proposes the fusion device processing pipeline consisting of a novel radar object detection and classification algorithm, State-of-the-Art camera processing algorithms, and a practical fusion algorithm to fuse the result from the camera and the radar. The proposed radar detection algorithm includes a novel clustering algorithm based on DBSCAN and a feature-based object classifier. The proposed algorithms show higher accuracy compared to the baseline. The camera processing algorithms include Yolov5 and StrongSort, which are pre-trained on their respective dataset and show high accuracy without the need for transfer learning. Finally, the practical fusion algorithm fuses the information between the radar and the camera at the decision level, where the camera results are matched with the radar results based on probability. The fusion allows the device to combine the high data association accuracy of the camera sensor with the additional measured states of the radar system to form a better understanding of the observed objects.