OPEMI: Online Performance Evaluation Metrics Index for Deep Learning-Based Autonomous Vehicles
- Title
- OPEMI: Online Performance Evaluation Metrics Index for Deep Learning-Based Autonomous Vehicles
- Author
- 남해운
- Keywords
- Autonomous driving; deep learning; end-to-end learning; online; offline performance metrics
- Issue Date
- 2023-02
- Publisher
- IEEE-INST ELECTRICAL ELECTRONICS ENGINEERS INC
- Citation
- IEEE ACCESS, v. 11, Page. 16951-16963
- Abstract
- Vision-based autonomous driving is rapidly growing. There are, however, presently no agreed-upon metrics for assessing how well deep neural network (DNN) models perform in driving. To compare novel approaches and architectures to existing ones, some researchers employed a mean error between labeled and predicted values in a test dataset and others presented a new metric that is designed to match their requirements. The discrepancy in the usage of various performance metrics and lack of objective metrics to judge the driving performance were our primary motives for developing a feasible solution. In this study, we propose online performance evaluation metrics index (OPEMI), an integrated metric that can evaluate the driving capabilities of autonomous driving models in various driving scenarios. To evaluate driving performance precisely and objectively, OPEMI incorporates several variables, including driving control stability, driving trajectory stability, journey duration, travel distance, success rate, and speed. To demonstrate the validity of OPEMI, we first confirmed that the prediction accuracy has a weak correlation with driving performance. Then, we have discussed the constraints in the existing driving performance metrics in certain circumstances, and their failure to assess the driving models. Finally, we conducted experiments with four popular DNN models and two in-house models under three different driving scenarios (generic, urban, and racing). The results show that the proposed evaluation metric, OPEMI, realistically displays driving performance and demonstrates its validity in various driving scenarios.
- URI
- https://ieeexplore.ieee.org/document/10047917https://repository.hanyang.ac.kr/handle/20.500.11754/185237
- ISSN
- 2169-3536
- DOI
- 10.1109/ACCESS.2023.3246104
- Appears in Collections:
- COLLEGE OF ENGINEERING SCIENCES[E](공학대학) > ELECTRICAL ENGINEERING(전자공학부) > Articles
- Files in This Item:
- 99287_남해운.pdfDownload
- Export
- RIS (EndNote)
- XLS (Excel)
- XML