Recently, deep learning has become increasingly popular in the field of vision based object classification, detection, and recognition as well as image generation. This dissertation deals with camera based traffic light (TL) recognition and light detection and ranging (LiDAR) based color image generation methods using deep learning networks for automotive vehicle systems.
In this study of the deep leaning based TL recognition, first of all, We have analyzed what color space is efficient. Six color spaces and three ensemble network models are applied and analyzed. Our simulation results show that the best performance is achieved with the combination method of RGB color space and Faster R-CNN with Inception-Resnet-v2. However, the conventional method has limited performance in several traffic lights with both small size and types of yellow, green-left, and off. To solve the problem, novel two-staged deep learning based TL recognition methods are proposed. To efficiently reduce the number of weight parameters and computational complexity, semantic segmentation technique and fully convolutional network (FCN) are applied. A binary-semantic segmentation network is proposed to detect small size TLs. We also propose a novel TL classification network including a convolution layer with three filters of (1×1). The simulation results show that the proposed TL recognition method outperforms the conventional Faster R-CNN network model with Inception-Resnet-v2 in terms of recognition performance, and it remarkably reduces the computational complexity and hardware requirements. The TL recognition method achieves up to 44.5% in overall mAP and 70.16% in mAP@0.5. Especially, the empirical results show that the proposed method gives great improvement for the detection and recognition of small TLs. The proposed method can also be implemented in real-time processing with the sacrifice of a minor decrease in recognition performance.
In the study of the deep leaning based color image generation from Lidar data, we propose a color image generation method from LiDAR 3D reflection intensity. The proposed method consists of 3D-to-2D projection and an image generation network (IGN). For the IGN, symmetric and asymmetric structured FCN are compared. Especially, an asymmetrically structured FCN is designed considering the sparseness of the projected reflection image. Through simulations, it is shown that the proposed method generates fairly good visual quality of images while maintaining almost the same color as the ground truth image. In particular, the asymmetrically structured FCN with a deeper decoder than encoder generates a higher-quality color image. Until the total number of layers reaches a certain number, the quality of the generated image monotonically increases. The proposed asymmetric FCN based color IGN model achieves up to 19.38 dB in peak signal-to-noise ratio and 0.5 in structural similarity index. In addition, the proposed IGN can generate shadow-free color images from LiDAR sensor data. We expect that the proposed method can generate daytime color images at night because the same LiDAR data can be obtained whether it is day or night.