Drowsiness Detection Based on Yawning Using Modified Pre-trained Model MobileNetV2 and ResNet50
Keywords:
Drowsiness detection, Facial-expression change, MobileNetV2, ResNet50, Yawn detection
Abstract
Traffic accidents are fatal events that need special attention. According to research by the National Transportation Safety Committee, 80% of traffic accidents are caused by human error, one of which is tired and drowsy drivers. The brain can interpret the vital fatigue of a drowsy driver sign as yawning. Therefore, yawning detection for preventing drowsy drivers’ imprudent can be developed using computer vision. This method is easy to implement and does not affect the driver when handling a vehicle. The research aimed to detect drowsy drivers based on facial expression changes of yawning by combining the Haar Cascade classifier and a modified pre-trained model, MobileNetV2 and ResNet50. Both proposed models accurately detected real-time images using a camera. The analysis showed that the yawning detection model based on the ResNet50 algorithm is more reliable, with the model obtaining 99% of accuracy. Furthermore, ResNet50 demonstrated reproducible outcomes for yawning detection, considering having good training capabilities and overall evaluation results.
Downloads
Download data is not yet available.
References
[1] L. Salvati, M. D’amore, A. Fiorentino, A. Pellegrino, P. Sena, and F. Villecco, “On-Road Detection of Driver Fatigue and
Drowsiness during Medium-Distance Journeys,” Entropy, vol. 23, no. 2, pp. 1–12, feb 2021.
[2] Z. Liu, Y. Peng, and W. Hu, “Driver Fatigue Detection Based on Deeply-Learned Facial Expression Representation,” Journal
of Visual Communication and Image Representation, vol. 71, p. 102723, aug 2020.
[3] Y. Ma, B. Chen, R. Li, C.Wang, J.Wang, Q. She, Z. Luo, and Y. Zhang, “Driving Fatigue Detection from EEG Using a Modified
PCANet Method,” Computational Intelligence and Neuroscience, vol. 2019, pp. 1–9, jul 2019.
[4] Z. Liu,W.Wei, H.Wang, Y. Zhang, Q. Zhang, and S. Li, “Intrusion Detection Based on Parallel Intelligent Optimization Feature
Extraction and Distributed Fuzzy Clustering in WSNs,” IEEE Access, vol. 6, no. November, pp. 72 201–72 211, 2018.
[5] J. Xi, S.Wang, T. Ding, J. Tian, H. Shao, and X. Miao, “Detection Model on Fatigue Driving Behaviors Based on the Operating
Parameters of Freight Vehicles,” Applied Sciences, vol. 11, no. 15, pp. 1–16, aug 2021.
[6] C. B. S. Maior, M. J. d. C. Moura, J. M. M. Santana, and I. D. Lins, “Real-time classification for autonomous drowsiness
detection using eye aspect ratio,” Expert Systems with Applications, vol. 158, p. 113505, nov 2020.
[7] C. Ryan, B. O’Sullivan, A. Elrasad, A. Cahill, J. Lemley, P. Kielty, C. Posch, and E. Perot, “Real-Time Face & Eye Tracking
and Blink Detection Using Event Cameras,” Neural Networks, vol. 141, no. 2021, pp. 87–97, 2021.
[8] H. Zidny Ilmadina, D. Apriliani, and D. S. Wibowo, “Deteksi Pengendara Mengantuk dengan Kombinasi Haar Cascade Classifier
dan Support Vector Machine,” Jurnal Informatika : Jurnal Pengembangan IT (JPIT), vol. 7, no. 1, pp. 1–7, 2022.
[9] C. Jacob´e de Naurois, C. Bourdin, A. Stratulat, E. Diaz, and J. L. Vercher, “Detection and Prediction of Driver Drowsiness
Using Artificial Neural Network Models,” Accident Analysis and Prevention, vol. 126, no. July, pp. 95–104, 2019.
[10] S. Rajamohana, E. Radhika, S. Priya, and S. Sangeetha, “Driver drowsiness detection system using hybrid approach of convolutional
neural network and bidirectional long short term memory (CNN BILSTM),” Materials Today: Proceedings, vol. 45,
no. 2, pp. 2897–2901, 2021.
[11] M. Knapik and B. Cyganek, “Driver’s fatigue recognition based on yawn detection in thermal images,” Neurocomputing, vol.
338, pp. 274–292, apr 2019.
[12] M. Kim and J. Koo, “Embedded System Performance Analysis for Implementing a Portable Drowsiness Detection System for
Drivers,” Technologies, vol. 11, no. 1, p. 124, dec 2022.
[13] K. Chen, T. Zhu, S. Li, and Y. Shi, “Real-time Yawning Detection Based on Machine Learning Algorithm and Time Series
Classification using Facial Feature Points,” in 2021 International Conference on High Performance Big Data and Intelligent
Systems (HPBD&IS). IEEE, dec 2021, pp. 276–280.
[14] A. Rahman, M. B. H. Hriday, and R. Khan, “Computer Vision-Based Approach to Detect Fatigue Driving and Face Mask for
Edge Computing Device,” Heliyon, vol. 8, no. 10, pp. 1–16, oct 2022.
[15] D. Vazquez, “Yawn Dataset— Kaggle.” [Online]. Available: https://www.kaggle.com/datasets/davidvazquezcic/yawn-dataset
[16] R. Raj, P. Rajiv, P. Kumar, M. Khari, E. Verd´u, R. G. Crespo, and G. Manogaran, “Feature based video stabilization based on
boosted HAAR Cascade and representative point matching algorithm,” Image and Vision Computing, vol. 101, p. 103957, sep
2020.
[17] D. Yang, B. Peng, Z. Al-Huda, A. Malik, and D. Zhai, “An Overview of Edge and Object Contour Detection,” Neurocomputing,
vol. 488, no. June, pp. 470–493, jun 2022.
[18] M. Sandler, A. Howard, M. Zhu, and A. Zhmoginov, “MobileNetV2: Inverted Residuals and Linear Bottlenecks,” in Proceedings
of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR),, 2018, pp. 4510–4520.
[19] K. He, X. Zhang, S. Ren, and J. Sun, “Deep Residual Learning for Image Recognition,” in 2016 IEEE Conference on Computer
Vision and Pattern Recognition (CVPR), 2016, pp. 770–778.
[20] A. Michele, V. Colin, and D. D. Santika, “MobileNet Convolutional Neural Networks and Support Vector Machines for Palmprint
Recognition,” Procedia Computer Science, vol. 157, pp. 110–117, 2019.
[21] M. H. Rahman, M. K. A. Jannat, M. S. Islam, G. Grossi, S. Bursic, and M. Aktaruzzaman, “Real-time face mask position
recognition system based on MobileNet model,” Smart Health, vol. 28, no. June, pp. 1–12, jun 2023.
[22] I. Shafi, A. Mazahir, A. Fatima, and I. Ashraf, “Internal Defects Detection and Classification in Hollow Cylindrical Surfaces
Using Single Shot Detection and MobileNet,” Measurement, vol. 202, p. 111836, oct 2022.
[23] W. Qi, “Object detection in high resolution optical image based on deep learning technique,” Natural Hazards Research, vol. 2,
no. 4, pp. 384–392, dec 2022.
[24] D. Sarwinda, R. H. Paradisa, A. Bustamam, and P. Anggia, “Deep Learning in Image Classification using Residual Network
(ResNet) Variants for Detection of Colorectal Cancer,” Procedia Computer Science, vol. 179, pp. 423–431, 2021.
[25] J. Chai, H. Zeng, A. Li, and E. W. Ngai, “Deep Learning in Computer Vision: A Critical Review of Emerging Techniques and
Application Scenarios,” Machine Learning with Applications, vol. 6, p. 100134, dec 2021.
[26] C. H. Karadal, M. C. Kaya, T. Tuncer, S. Dogan, and U. R. Acharya, “Automated classification of remote sensing images using
multileveled MobileNetV2 and DWT techniques,” Expert Systems with Applications, vol. 185, pp. 1–12, dec 2021.
Drowsiness during Medium-Distance Journeys,” Entropy, vol. 23, no. 2, pp. 1–12, feb 2021.
[2] Z. Liu, Y. Peng, and W. Hu, “Driver Fatigue Detection Based on Deeply-Learned Facial Expression Representation,” Journal
of Visual Communication and Image Representation, vol. 71, p. 102723, aug 2020.
[3] Y. Ma, B. Chen, R. Li, C.Wang, J.Wang, Q. She, Z. Luo, and Y. Zhang, “Driving Fatigue Detection from EEG Using a Modified
PCANet Method,” Computational Intelligence and Neuroscience, vol. 2019, pp. 1–9, jul 2019.
[4] Z. Liu,W.Wei, H.Wang, Y. Zhang, Q. Zhang, and S. Li, “Intrusion Detection Based on Parallel Intelligent Optimization Feature
Extraction and Distributed Fuzzy Clustering in WSNs,” IEEE Access, vol. 6, no. November, pp. 72 201–72 211, 2018.
[5] J. Xi, S.Wang, T. Ding, J. Tian, H. Shao, and X. Miao, “Detection Model on Fatigue Driving Behaviors Based on the Operating
Parameters of Freight Vehicles,” Applied Sciences, vol. 11, no. 15, pp. 1–16, aug 2021.
[6] C. B. S. Maior, M. J. d. C. Moura, J. M. M. Santana, and I. D. Lins, “Real-time classification for autonomous drowsiness
detection using eye aspect ratio,” Expert Systems with Applications, vol. 158, p. 113505, nov 2020.
[7] C. Ryan, B. O’Sullivan, A. Elrasad, A. Cahill, J. Lemley, P. Kielty, C. Posch, and E. Perot, “Real-Time Face & Eye Tracking
and Blink Detection Using Event Cameras,” Neural Networks, vol. 141, no. 2021, pp. 87–97, 2021.
[8] H. Zidny Ilmadina, D. Apriliani, and D. S. Wibowo, “Deteksi Pengendara Mengantuk dengan Kombinasi Haar Cascade Classifier
dan Support Vector Machine,” Jurnal Informatika : Jurnal Pengembangan IT (JPIT), vol. 7, no. 1, pp. 1–7, 2022.
[9] C. Jacob´e de Naurois, C. Bourdin, A. Stratulat, E. Diaz, and J. L. Vercher, “Detection and Prediction of Driver Drowsiness
Using Artificial Neural Network Models,” Accident Analysis and Prevention, vol. 126, no. July, pp. 95–104, 2019.
[10] S. Rajamohana, E. Radhika, S. Priya, and S. Sangeetha, “Driver drowsiness detection system using hybrid approach of convolutional
neural network and bidirectional long short term memory (CNN BILSTM),” Materials Today: Proceedings, vol. 45,
no. 2, pp. 2897–2901, 2021.
[11] M. Knapik and B. Cyganek, “Driver’s fatigue recognition based on yawn detection in thermal images,” Neurocomputing, vol.
338, pp. 274–292, apr 2019.
[12] M. Kim and J. Koo, “Embedded System Performance Analysis for Implementing a Portable Drowsiness Detection System for
Drivers,” Technologies, vol. 11, no. 1, p. 124, dec 2022.
[13] K. Chen, T. Zhu, S. Li, and Y. Shi, “Real-time Yawning Detection Based on Machine Learning Algorithm and Time Series
Classification using Facial Feature Points,” in 2021 International Conference on High Performance Big Data and Intelligent
Systems (HPBD&IS). IEEE, dec 2021, pp. 276–280.
[14] A. Rahman, M. B. H. Hriday, and R. Khan, “Computer Vision-Based Approach to Detect Fatigue Driving and Face Mask for
Edge Computing Device,” Heliyon, vol. 8, no. 10, pp. 1–16, oct 2022.
[15] D. Vazquez, “Yawn Dataset— Kaggle.” [Online]. Available: https://www.kaggle.com/datasets/davidvazquezcic/yawn-dataset
[16] R. Raj, P. Rajiv, P. Kumar, M. Khari, E. Verd´u, R. G. Crespo, and G. Manogaran, “Feature based video stabilization based on
boosted HAAR Cascade and representative point matching algorithm,” Image and Vision Computing, vol. 101, p. 103957, sep
2020.
[17] D. Yang, B. Peng, Z. Al-Huda, A. Malik, and D. Zhai, “An Overview of Edge and Object Contour Detection,” Neurocomputing,
vol. 488, no. June, pp. 470–493, jun 2022.
[18] M. Sandler, A. Howard, M. Zhu, and A. Zhmoginov, “MobileNetV2: Inverted Residuals and Linear Bottlenecks,” in Proceedings
of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR),, 2018, pp. 4510–4520.
[19] K. He, X. Zhang, S. Ren, and J. Sun, “Deep Residual Learning for Image Recognition,” in 2016 IEEE Conference on Computer
Vision and Pattern Recognition (CVPR), 2016, pp. 770–778.
[20] A. Michele, V. Colin, and D. D. Santika, “MobileNet Convolutional Neural Networks and Support Vector Machines for Palmprint
Recognition,” Procedia Computer Science, vol. 157, pp. 110–117, 2019.
[21] M. H. Rahman, M. K. A. Jannat, M. S. Islam, G. Grossi, S. Bursic, and M. Aktaruzzaman, “Real-time face mask position
recognition system based on MobileNet model,” Smart Health, vol. 28, no. June, pp. 1–12, jun 2023.
[22] I. Shafi, A. Mazahir, A. Fatima, and I. Ashraf, “Internal Defects Detection and Classification in Hollow Cylindrical Surfaces
Using Single Shot Detection and MobileNet,” Measurement, vol. 202, p. 111836, oct 2022.
[23] W. Qi, “Object detection in high resolution optical image based on deep learning technique,” Natural Hazards Research, vol. 2,
no. 4, pp. 384–392, dec 2022.
[24] D. Sarwinda, R. H. Paradisa, A. Bustamam, and P. Anggia, “Deep Learning in Image Classification using Residual Network
(ResNet) Variants for Detection of Colorectal Cancer,” Procedia Computer Science, vol. 179, pp. 423–431, 2021.
[25] J. Chai, H. Zeng, A. Li, and E. W. Ngai, “Deep Learning in Computer Vision: A Critical Review of Emerging Techniques and
Application Scenarios,” Machine Learning with Applications, vol. 6, p. 100134, dec 2021.
[26] C. H. Karadal, M. C. Kaya, T. Tuncer, S. Dogan, and U. R. Acharya, “Automated classification of remote sensing images using
multileveled MobileNetV2 and DWT techniques,” Expert Systems with Applications, vol. 185, pp. 1–12, dec 2021.
Published
2023-06-06
How to Cite
Ilmadina, H., Naufal, M., & Wibowo, D. (2023). Drowsiness Detection Based on Yawning Using Modified Pre-trained Model MobileNetV2 and ResNet50. MATRIK : Jurnal Manajemen, Teknik Informatika Dan Rekayasa Komputer, 22(3), 419-430. https://doi.org/https://doi.org/10.30812/matrik.v22i3.2785
Issue
Section
Articles
This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.
.png)
