Drowsiness Detection Based on Yawning Using Modified Pre-trained Model MobileNetV2 and ResNet50
DOI:
https://doi.org/10.30812/matrik.v22i3.2785Keywords:
Drowsiness detection, Facial-expression change, MobileNetV2, ResNet50, Yawn detectionAbstract
Traffic accidents are fatal events that need special attention. According to research by the National Transportation Safety Committee, 80% of traffic accidents are caused by human error, one of which is tired and drowsy drivers. The brain can interpret the vital fatigue of a drowsy driver sign as yawning. Therefore, yawning detection for preventing drowsy drivers’ imprudent can be developed using computer vision. This method is easy to implement and does not affect the driver when handling a vehicle. The research aimed to detect drowsy drivers based on facial expression changes of yawning by combining the Haar Cascade classifier and a modified pre-trained model, MobileNetV2 and ResNet50. Both proposed models accurately detected real-time images using a camera. The analysis showed that the yawning detection model based on the ResNet50 algorithm is more reliable, with the model obtaining 99% of accuracy. Furthermore, ResNet50 demonstrated reproducible outcomes for yawning detection, considering having good training capabilities and overall evaluation results.
Downloads
Download data is not yet available.
References
[1] L. Salvati, M. D’amore, A. Fiorentino, A. Pellegrino, P. Sena, and F. Villecco, “On-Road Detection of Driver Fatigue and
Drowsiness during Medium-Distance Journeys,†Entropy, vol. 23, no. 2, pp. 1–12, feb 2021.
[2] Z. Liu, Y. Peng, and W. Hu, “Driver Fatigue Detection Based on Deeply-Learned Facial Expression Representation,†Journal
of Visual Communication and Image Representation, vol. 71, p. 102723, aug 2020.
[3] Y. Ma, B. Chen, R. Li, C.Wang, J.Wang, Q. She, Z. Luo, and Y. Zhang, “Driving Fatigue Detection from EEG Using a Modified
PCANet Method,†Computational Intelligence and Neuroscience, vol. 2019, pp. 1–9, jul 2019.
[4] Z. Liu,W.Wei, H.Wang, Y. Zhang, Q. Zhang, and S. Li, “Intrusion Detection Based on Parallel Intelligent Optimization Feature
Extraction and Distributed Fuzzy Clustering in WSNs,†IEEE Access, vol. 6, no. November, pp. 72 201–72 211, 2018.
[5] J. Xi, S.Wang, T. Ding, J. Tian, H. Shao, and X. Miao, “Detection Model on Fatigue Driving Behaviors Based on the Operating
Parameters of Freight Vehicles,†Applied Sciences, vol. 11, no. 15, pp. 1–16, aug 2021.
[6] C. B. S. Maior, M. J. d. C. Moura, J. M. M. Santana, and I. D. Lins, “Real-time classification for autonomous drowsiness
detection using eye aspect ratio,†Expert Systems with Applications, vol. 158, p. 113505, nov 2020.
[7] C. Ryan, B. O’Sullivan, A. Elrasad, A. Cahill, J. Lemley, P. Kielty, C. Posch, and E. Perot, “Real-Time Face & Eye Tracking
and Blink Detection Using Event Cameras,†Neural Networks, vol. 141, no. 2021, pp. 87–97, 2021.
[8] H. Zidny Ilmadina, D. Apriliani, and D. S. Wibowo, “Deteksi Pengendara Mengantuk dengan Kombinasi Haar Cascade Classifier
dan Support Vector Machine,†Jurnal Informatika : Jurnal Pengembangan IT (JPIT), vol. 7, no. 1, pp. 1–7, 2022.
[9] C. Jacob´e de Naurois, C. Bourdin, A. Stratulat, E. Diaz, and J. L. Vercher, “Detection and Prediction of Driver Drowsiness
Using Artificial Neural Network Models,†Accident Analysis and Prevention, vol. 126, no. July, pp. 95–104, 2019.
[10] S. Rajamohana, E. Radhika, S. Priya, and S. Sangeetha, “Driver drowsiness detection system using hybrid approach of convolutional
neural network and bidirectional long short term memory (CNN BILSTM),†Materials Today: Proceedings, vol. 45,
no. 2, pp. 2897–2901, 2021.
[11] M. Knapik and B. Cyganek, “Driver’s fatigue recognition based on yawn detection in thermal images,†Neurocomputing, vol.
338, pp. 274–292, apr 2019.
[12] M. Kim and J. Koo, “Embedded System Performance Analysis for Implementing a Portable Drowsiness Detection System for
Drivers,†Technologies, vol. 11, no. 1, p. 124, dec 2022.
[13] K. Chen, T. Zhu, S. Li, and Y. Shi, “Real-time Yawning Detection Based on Machine Learning Algorithm and Time Series
Classification using Facial Feature Points,†in 2021 International Conference on High Performance Big Data and Intelligent
Systems (HPBD&IS). IEEE, dec 2021, pp. 276–280.
[14] A. Rahman, M. B. H. Hriday, and R. Khan, “Computer Vision-Based Approach to Detect Fatigue Driving and Face Mask for
Edge Computing Device,†Heliyon, vol. 8, no. 10, pp. 1–16, oct 2022.
[15] D. Vazquez, “Yawn Dataset— Kaggle.†[Online]. Available: https://www.kaggle.com/datasets/davidvazquezcic/yawn-dataset
[16] R. Raj, P. Rajiv, P. Kumar, M. Khari, E. Verd´u, R. G. Crespo, and G. Manogaran, “Feature based video stabilization based on
boosted HAAR Cascade and representative point matching algorithm,†Image and Vision Computing, vol. 101, p. 103957, sep
2020.
[17] D. Yang, B. Peng, Z. Al-Huda, A. Malik, and D. Zhai, “An Overview of Edge and Object Contour Detection,†Neurocomputing,
vol. 488, no. June, pp. 470–493, jun 2022.
[18] M. Sandler, A. Howard, M. Zhu, and A. Zhmoginov, “MobileNetV2: Inverted Residuals and Linear Bottlenecks,†in Proceedings
of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR),, 2018, pp. 4510–4520.
[19] K. He, X. Zhang, S. Ren, and J. Sun, “Deep Residual Learning for Image Recognition,†in 2016 IEEE Conference on Computer
Vision and Pattern Recognition (CVPR), 2016, pp. 770–778.
[20] A. Michele, V. Colin, and D. D. Santika, “MobileNet Convolutional Neural Networks and Support Vector Machines for Palmprint
Recognition,†Procedia Computer Science, vol. 157, pp. 110–117, 2019.
[21] M. H. Rahman, M. K. A. Jannat, M. S. Islam, G. Grossi, S. Bursic, and M. Aktaruzzaman, “Real-time face mask position
recognition system based on MobileNet model,†Smart Health, vol. 28, no. June, pp. 1–12, jun 2023.
[22] I. Shafi, A. Mazahir, A. Fatima, and I. Ashraf, “Internal Defects Detection and Classification in Hollow Cylindrical Surfaces
Using Single Shot Detection and MobileNet,†Measurement, vol. 202, p. 111836, oct 2022.
[23] W. Qi, “Object detection in high resolution optical image based on deep learning technique,†Natural Hazards Research, vol. 2,
no. 4, pp. 384–392, dec 2022.
[24] D. Sarwinda, R. H. Paradisa, A. Bustamam, and P. Anggia, “Deep Learning in Image Classification using Residual Network
(ResNet) Variants for Detection of Colorectal Cancer,†Procedia Computer Science, vol. 179, pp. 423–431, 2021.
[25] J. Chai, H. Zeng, A. Li, and E. W. Ngai, “Deep Learning in Computer Vision: A Critical Review of Emerging Techniques and
Application Scenarios,†Machine Learning with Applications, vol. 6, p. 100134, dec 2021.
[26] C. H. Karadal, M. C. Kaya, T. Tuncer, S. Dogan, and U. R. Acharya, “Automated classification of remote sensing images using
multileveled MobileNetV2 and DWT techniques,†Expert Systems with Applications, vol. 185, pp. 1–12, dec 2021.
Drowsiness during Medium-Distance Journeys,†Entropy, vol. 23, no. 2, pp. 1–12, feb 2021.
[2] Z. Liu, Y. Peng, and W. Hu, “Driver Fatigue Detection Based on Deeply-Learned Facial Expression Representation,†Journal
of Visual Communication and Image Representation, vol. 71, p. 102723, aug 2020.
[3] Y. Ma, B. Chen, R. Li, C.Wang, J.Wang, Q. She, Z. Luo, and Y. Zhang, “Driving Fatigue Detection from EEG Using a Modified
PCANet Method,†Computational Intelligence and Neuroscience, vol. 2019, pp. 1–9, jul 2019.
[4] Z. Liu,W.Wei, H.Wang, Y. Zhang, Q. Zhang, and S. Li, “Intrusion Detection Based on Parallel Intelligent Optimization Feature
Extraction and Distributed Fuzzy Clustering in WSNs,†IEEE Access, vol. 6, no. November, pp. 72 201–72 211, 2018.
[5] J. Xi, S.Wang, T. Ding, J. Tian, H. Shao, and X. Miao, “Detection Model on Fatigue Driving Behaviors Based on the Operating
Parameters of Freight Vehicles,†Applied Sciences, vol. 11, no. 15, pp. 1–16, aug 2021.
[6] C. B. S. Maior, M. J. d. C. Moura, J. M. M. Santana, and I. D. Lins, “Real-time classification for autonomous drowsiness
detection using eye aspect ratio,†Expert Systems with Applications, vol. 158, p. 113505, nov 2020.
[7] C. Ryan, B. O’Sullivan, A. Elrasad, A. Cahill, J. Lemley, P. Kielty, C. Posch, and E. Perot, “Real-Time Face & Eye Tracking
and Blink Detection Using Event Cameras,†Neural Networks, vol. 141, no. 2021, pp. 87–97, 2021.
[8] H. Zidny Ilmadina, D. Apriliani, and D. S. Wibowo, “Deteksi Pengendara Mengantuk dengan Kombinasi Haar Cascade Classifier
dan Support Vector Machine,†Jurnal Informatika : Jurnal Pengembangan IT (JPIT), vol. 7, no. 1, pp. 1–7, 2022.
[9] C. Jacob´e de Naurois, C. Bourdin, A. Stratulat, E. Diaz, and J. L. Vercher, “Detection and Prediction of Driver Drowsiness
Using Artificial Neural Network Models,†Accident Analysis and Prevention, vol. 126, no. July, pp. 95–104, 2019.
[10] S. Rajamohana, E. Radhika, S. Priya, and S. Sangeetha, “Driver drowsiness detection system using hybrid approach of convolutional
neural network and bidirectional long short term memory (CNN BILSTM),†Materials Today: Proceedings, vol. 45,
no. 2, pp. 2897–2901, 2021.
[11] M. Knapik and B. Cyganek, “Driver’s fatigue recognition based on yawn detection in thermal images,†Neurocomputing, vol.
338, pp. 274–292, apr 2019.
[12] M. Kim and J. Koo, “Embedded System Performance Analysis for Implementing a Portable Drowsiness Detection System for
Drivers,†Technologies, vol. 11, no. 1, p. 124, dec 2022.
[13] K. Chen, T. Zhu, S. Li, and Y. Shi, “Real-time Yawning Detection Based on Machine Learning Algorithm and Time Series
Classification using Facial Feature Points,†in 2021 International Conference on High Performance Big Data and Intelligent
Systems (HPBD&IS). IEEE, dec 2021, pp. 276–280.
[14] A. Rahman, M. B. H. Hriday, and R. Khan, “Computer Vision-Based Approach to Detect Fatigue Driving and Face Mask for
Edge Computing Device,†Heliyon, vol. 8, no. 10, pp. 1–16, oct 2022.
[15] D. Vazquez, “Yawn Dataset— Kaggle.†[Online]. Available: https://www.kaggle.com/datasets/davidvazquezcic/yawn-dataset
[16] R. Raj, P. Rajiv, P. Kumar, M. Khari, E. Verd´u, R. G. Crespo, and G. Manogaran, “Feature based video stabilization based on
boosted HAAR Cascade and representative point matching algorithm,†Image and Vision Computing, vol. 101, p. 103957, sep
2020.
[17] D. Yang, B. Peng, Z. Al-Huda, A. Malik, and D. Zhai, “An Overview of Edge and Object Contour Detection,†Neurocomputing,
vol. 488, no. June, pp. 470–493, jun 2022.
[18] M. Sandler, A. Howard, M. Zhu, and A. Zhmoginov, “MobileNetV2: Inverted Residuals and Linear Bottlenecks,†in Proceedings
of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR),, 2018, pp. 4510–4520.
[19] K. He, X. Zhang, S. Ren, and J. Sun, “Deep Residual Learning for Image Recognition,†in 2016 IEEE Conference on Computer
Vision and Pattern Recognition (CVPR), 2016, pp. 770–778.
[20] A. Michele, V. Colin, and D. D. Santika, “MobileNet Convolutional Neural Networks and Support Vector Machines for Palmprint
Recognition,†Procedia Computer Science, vol. 157, pp. 110–117, 2019.
[21] M. H. Rahman, M. K. A. Jannat, M. S. Islam, G. Grossi, S. Bursic, and M. Aktaruzzaman, “Real-time face mask position
recognition system based on MobileNet model,†Smart Health, vol. 28, no. June, pp. 1–12, jun 2023.
[22] I. Shafi, A. Mazahir, A. Fatima, and I. Ashraf, “Internal Defects Detection and Classification in Hollow Cylindrical Surfaces
Using Single Shot Detection and MobileNet,†Measurement, vol. 202, p. 111836, oct 2022.
[23] W. Qi, “Object detection in high resolution optical image based on deep learning technique,†Natural Hazards Research, vol. 2,
no. 4, pp. 384–392, dec 2022.
[24] D. Sarwinda, R. H. Paradisa, A. Bustamam, and P. Anggia, “Deep Learning in Image Classification using Residual Network
(ResNet) Variants for Detection of Colorectal Cancer,†Procedia Computer Science, vol. 179, pp. 423–431, 2021.
[25] J. Chai, H. Zeng, A. Li, and E. W. Ngai, “Deep Learning in Computer Vision: A Critical Review of Emerging Techniques and
Application Scenarios,†Machine Learning with Applications, vol. 6, p. 100134, dec 2021.
[26] C. H. Karadal, M. C. Kaya, T. Tuncer, S. Dogan, and U. R. Acharya, “Automated classification of remote sensing images using
multileveled MobileNetV2 and DWT techniques,†Expert Systems with Applications, vol. 185, pp. 1–12, dec 2021.
Downloads
Published
2023-06-06
Issue
Section
Articles
How to Cite
[1]
H. Z. Ilmadina, M. Naufal, and D. S. Wibowo, “Drowsiness Detection Based on Yawning Using Modified Pre-trained Model MobileNetV2 and ResNet50”, MATRIK, vol. 22, no. 3, pp. 419–430, Jun. 2023, doi: 10.30812/matrik.v22i3.2785.
Similar Articles
- Rizky Hafizh Jatmiko, Yoga Pristyanto, Investigating The Effectiveness of Various Convolutional Neural Network Model Architectures for Skin Cancer Melanoma Classification , MATRIK : Jurnal Manajemen, Teknik Informatika dan Rekayasa Komputer: Vol. 23 No. 1 (2023)
- Bambang Krismono Triwijoyo, SEGMENTASI CITRA PEMBULUH DARAH RETINA MENGGUNAKAN METODE DETEKSI GARIS MULTI SKALA , MATRIK : Jurnal Manajemen, Teknik Informatika dan Rekayasa Komputer: Vol. 15 No. 1 (2015)
- Ni Gusti Ayu Dasriani, Sirojul Hadi, Moch Syahrir, Intelligent System for Internet of Things-Based Building Fire Safety with Naive Bayes Algorithm , MATRIK : Jurnal Manajemen, Teknik Informatika dan Rekayasa Komputer: Vol. 23 No. 1 (2023)
- Syafri Arlis, Muhammad Reza Putra, Musli Yanto, Improved Image Segmentation using Adaptive Threshold Morphology on CT-Scan Images for Brain Tumor Detection , MATRIK : Jurnal Manajemen, Teknik Informatika dan Rekayasa Komputer: Vol. 23 No. 3 (2024)
- Mukhlis Mukhlis, Puput Yuniar Maulidia, Achmad Mujib, Adi Muhajirin, Alpi Surya Perdana, Integration of Deep Learning and Autoregressive Models for Marine Data Prediction , MATRIK : Jurnal Manajemen, Teknik Informatika dan Rekayasa Komputer: Vol. 24 No. 1 (2024)
- Aini Suri Talita, Aristiawan Wiguna, Implementasi Algoritma Long Short-Term Memory (LSTM) Untuk Mendeteksi Ujaran Kebencian (Hate Speech) Pada Kasus Pilpres 2019 , MATRIK : Jurnal Manajemen, Teknik Informatika dan Rekayasa Komputer: Vol. 19 No. 1 (2019)
- Mudafiq Riyan Pratama, Muhammad Yunus, Sistem Deteksi Struktur Kalimat Bahasa Arab Menggunakan Algoritma Light Stemming , MATRIK : Jurnal Manajemen, Teknik Informatika dan Rekayasa Komputer: Vol. 19 No. 1 (2019)
- Solikhun Solikhun, Lise Pujiastuti, Mochamad Wahyudi, Enhancing Lung Cancer Prediction Accuracy UsingQuantum-Enhanced K-Medoids with Manhattan Distance , MATRIK : Jurnal Manajemen, Teknik Informatika dan Rekayasa Komputer: Vol. 24 No. 3 (2025)
- sri suharti, Anton Yudhana, Imam Riadi, Forensik Jaringan DDoS menggunakan Metode ADDIE dan HIDS pada Sistem Operasi Proprietary , MATRIK : Jurnal Manajemen, Teknik Informatika dan Rekayasa Komputer: Vol. 21 No. 3 (2022)
- Supangat Supangat, Mohd Zainuri Bin Saringat, Mochamad Yovi Fatchur Rochman, Predicting Handling Covid-19 Opinion using Naive Bayes and TF-IDF for Polarity Detection , MATRIK : Jurnal Manajemen, Teknik Informatika dan Rekayasa Komputer: Vol. 22 No. 2 (2023)
You may also start an advanced similarity search for this article.
Most read articles by the same author(s)
- Dairoh Dairoh, Very Kurnia Bakti, Muhammad Naufal, Neural Network dan Particle Swam Optimization untuk Penunjang Keputusan Antipasi Mahasiswa Pra Lulus Bekerja Sesuai Bidang , MATRIK : Jurnal Manajemen, Teknik Informatika dan Rekayasa Komputer: Vol. 21 No. 1 (2021)
.png)
