DynamicWeighted Particle Swarm Optimization - Support Vector Machine Optimization in Recursive Feature Elimination Feature Selection
Optimization in Recursive Feature Elimination
Keywords:
Dynamic Weighted Particle Swarm, Feature Selection, Optimization, Recursive Feature Elimination, Support Vector Machine
Abstract
Feature Selection is a crucial step in data preprocessing to enhance machine learning efficiency, reduce computational complexity, and improve classification accuracy. The main challenge in feature selection for classification is identifying the most relevant and informative subset to enhance prediction accuracy. Previous studies often resulted in suboptimal subsets, leading to poor model performance and low accuracy. This research aims to enhance classification accuracy by utilizing Recursive Feature Elimination (RFE) combined with Dynamic Weighted Particle Swarm Optimization (DWPSO) and Support Vector Machine (SVM) algorithms. The research method involves the utilization of 12 datasets from the University of California, Irvine (UCI) repository, where features are selected via RFE and applied to the DWPSO-SVM algorithm. RFE iteratively removes the weakest features, constructing a model with the most relevant features to enhance accuracy. The research findings indicate that DWPSO-SVM with RFE significantly improves classification accuracy. For example, accuracy on the Breast Cancer dataset increased from 58% to 76%, and on the Heart dataset from 80% to 97%. The highest accuracy achieved was 100% on the Iris dataset. The conclusion of these findings that RFE in DWPSO-SVM offers consistent and balanced results in True Positive Rate (TPR) and True Negative Rate (TNR), providing reliable and accurate predictions for various applications.
Downloads
Download data is not yet available.
References
[1] N. Maurya, N. Kumar, V. Maurya, and V. Kumar Maurya, “a Review on Machine Learning (Feature Selection, Classification
and Clustering) Approaches of Big Data Mining in Different Area of Research Journal of Critical Reviews a Review on Machine
Learning (Feature Selection, Classification and Clustering) Approac,” Article in Journal of Critical Reviews, vol. 7, no. August,
p. 2020, 2020, https://doi.org/10.31838/jcr.07.19.322.
[2] N. Rtayli and N. Enneya, “Enhanced credit card fraud detection based on SVM-recursive feature elimination and hyperparameters
optimization,” Journal of Information Security and Applications, vol. 55, no. September, 2020, https://doi.org/
10.1016/j.jisa.2020.102596.
[3] Y. Yin, J. Jang-Jaccard, W. Xu, A. Singh, J. Zhu, F. Sabrina, and J. Kwak, “IGRF-RFE: a hybrid feature selection method for
MLP-based network intrusion detection on UNSW-NB15 dataset,” Journal of Big Data, vol. 10, no. 1, 2023, https://doi.org/10.
1186/s40537-023-00694-8.
[4] H. Jeon and S. Oh, “applied sciences Hybrid-Recursive Feature Elimination for E ffi cient Feature Selection,” Applied Sciences,
pp. 1–8, 2020.
[5] R. Zebari, A. Abdulazeez, D. Zeebaree, D. Zebari, and J. Saeed, “A Comprehensive Review of Dimensionality Reduction
Techniques for Feature Selection and Feature Extraction,” Journal of Applied Science and Technology Trends, vol. 1, no. 2, pp.
56–70, 2020, https://doi.org/10.38094/jastt1224.
[6] M. Ahsan, R. Gomes, M. M. Chowdhury, and K. E. Nygard, “Enhancing Machine Learning Prediction in Cybersecurity Using
Dynamic Feature Selector,” Journal of Cybersecurity and Privacy, vol. 1, no. 1, pp. 199–218, 2021, https://doi.org/10.3390/
jcp1010011.
[7] P. Saraswat, “Supervised Machine Learning Algorithm: A Review of Classification Techniques,” Smart Innovation, Systems
and Technologies, vol. 273, pp. 477–482, 2022, https://doi.org/10.1007/978-3-030-92905-3 58.
[8] J. Cervantes, F. Garcia-Lamont, L. Rodr´ıguez-Mazahua, and A. Lopez, “A comprehensive survey on support vector machine
classification: Applications, challenges and trends,” Neurocomputing, vol. 408, no. xxxx, pp. 189–215, 2020, https://doi.org/10.
1016/j.neucom.2019.10.118.
[9] B. R. Murlidhar, H. Nguyen, J. Rostami, X. N. Bui, D. J. Armaghani, P. Ragam, and E. T. Mohamad, “Prediction of flyrock
distance induced by mine blasting using a novel Harris Hawks optimization-based multi-layer perceptron neural network,”
Journal of Rock Mechanics and Geotechnical Engineering, vol. 13, no. 6, pp. 1413–1427, 2021, https://doi.org/10.1016/j.
jrmge.2021.08.005.
[10] S. M. Cherian and K. Rajitha, “Random forest and support vector machine classifiers for coastal wetland characterization using
the combination of features derived from optical data and synthetic aperture radar dataset,” Journal of Water and Climate
Change, vol. 15, no. 1, pp. 29–49, 2024, https://doi.org/10.2166/wcc.2023.238.
[11] E. Elgeldawi, A. Sayed, A. R. Galal, and A. M. Zaki, “Hyperparameter tuning for machine learning algorithms used for arabic
sentiment analysis,” Informatics, vol. 8, no. 4, pp. 1–21, 2021, https://doi.org/10.3390/informatics8040079.
[12] E. S. El-Kenawy and M. Eid, “Hybrid gray wolf and particle swarm optimization for feature selection,” International Journal
of Innovative Computing, Information and Control, vol. 16, no. 3, pp. 831–844, 2020, https://doi.org/10.24507/ijicic.16.03.831.
[13] M.Weiel, M. G¨otz, A. Klein, D. Coquelin, R. Floca, and A. Schug, “biomolecular simulation parameters with flexible objective
functions,” Nature Machine Intelligence, vol. 3, no. August, pp. 727–734, 2021, https://doi.org/10.1038/s42256-021-00366-3.
[14] E. M. Senan, M. H. Al-Adhaileh, F.W. Alsaade, T. H. Aldhyani, A. A. Alqarni, N. Alsharif, M. I. Uddin, A. H. Alahmadi, M. E.
Jadhav, and M. Y. Alzahrani, “Diagnosis of Chronic Kidney Disease Using Effective Classification Algorithms and Recursive
Feature Elimination Techniques,” Journal of Healthcare Engineering, vol. 2021, 2021, https://doi.org/10.1155/2021/1004767.
[15] J. Xia, L. Sun, S. Xu, Q. Xiang, J. Zhao, W. Xiong, Y. Xu, and S. Chu, “A model using support vector machines recursive
feature elimination (Svm-rfe) algorithm to classify whether copd patients have been continuously managed according to gold
guidelines,” International Journal of COPD, vol. 15, pp. 2779–2786, 2020, https://doi.org/10.2147/COPD.S271237.
[16] S. Zhao and Z. Zhao, “A Comparative Study of Landslide Susceptibility Mapping Using SVM and PSO-SVM Models Based
on Grid and Slope Units,” Mathematical Problems in Engineering, vol. 2021, 2021, https://doi.org/10.1155/2021/8854606.
[17] M. F. Akbar, M. I. Mazdadi, H. Saragih, and F. Abadi, “Implementation of Information Gain Ratio and Particle Swarm Optimization
in the Sentiment Analysis Classification of Covid-19 Vaccine Using Support Vector Machine,” vol. 5, no. 4, pp.
261–270, 2023.
[18] J. Wang, X. Wang, X. Li, and J. Yi, “A Hybrid Particle Swarm Optimization Algorithm with Dynamic Adjustment of Inertia
Weight Based on a New Feature Selection Method to Optimize SVM Parameters,” Entropy, vol. 25, no. 3, 2023, https://doi.org/
10.3390/e25030531.
[19] C. Aroniadi and G. N. Beligiannis, “Applying Particle Swarm Optimization Variations to Solve the Transportation Problem
Effectively,” Algorithms, vol. 16, no. 8, 2023, https://doi.org/10.3390/a16080372.
[20] A. Performance, E. Alshdaifat, D. Alshdaifat, A. Alsarhan, F. Hussein, and S. Moh, “The Effect of Preprocessing Techniques ,
Applied to Numeric,” Data, vol. 6, no. 11, 2021.
[21] M. Ramkumar Prabhu, R. Sivaraman, N. Nagabhooshanam, R. Sampath Kumar, and S. Sathish Salunkhe, “Empowering artificial
intelligence-based multi-biometric image sensor for human identification,” Measurement: Sensors, vol. 33, no. April, p.
101082, 2024, https://doi.org/10.1016/j.measen.2024.101082.
[22] U. M. Khaire and R. Dhanalakshmi, “Stability of feature selection algorithm: A review,” Journal of King Saud University -
Computer and Information Sciences, vol. 34, no. 4, pp. 1060–1073, 2022, https://doi.org/10.1016/j.jksuci.2019.06.012.
[23] K. Shah, H. Patel, D. Sanghvi, and M. Shah, “A Comparative Analysis of Logistic Regression , Random Forest and KNN
Models for the Text Classification,” Augmented Human Research, vol. 0, 2020, https://doi.org/10.1007/s41133-020-00032-0.
[24] J. Alcaraz, M. Labb´e, and M. Landete, “Support Vector Machine with feature selection : A multiobjective approach,” vol. 204,
no. September 2021, 2022.
[25] M. Pan, C. Li, R. Gao, Y. Huang, H. You, T. Gu, and F. Qin, “Photovoltaic power forecasting based on a support vector machine
with improved ant colony optimization,” Journal of Cleaner Production, vol. 277, 2020, https://doi.org/10.1016/j.jclepro.2020.
123948.
[26] M. Jain, V. Saihjpal, N. Singh, and S. B. Singh, “An Overview of Variants and Advancements of PSO Algorithm,” Applied
Sciences (Switzerland), vol. 12, no. 17, pp. 1–21, 2022, https://doi.org/10.3390/app12178392.
[27] H. Reza, R. Zaman, and F. Soleimanian, An improved particle swarm optimization with backtracking search optimization
algorithm for solving continuous optimization problems. Springer London, 2021, no. 0123456789, https://doi.org/10.1007/
s00366-021-01431-6.
[28] N. Van Tinh, “Enhanced Forecasting Accuracy of Fuzzy Time Series Model Based on Combined Fuzzy C-Mean Clustering
with Particle Swam Optimization,” International Journal of Computational Intelligence and Applications, vol. 19, no. 2, 2020,
https://doi.org/10.1142/S1469026820500170.
[29] S. Surono, K. W. Goh, C. W. Onn, A. Nurraihan, N. S. Siregar, A. B. Saeid, and T. T. Wijaya, “Optimization of Markov
Weighted Fuzzy Time Series Forecasting Using Genetic Algorithm (GA) and Particle Swarm Optimization (PSO),” Emerging
Science Journal, vol. 6, no. 6, pp. 1375–1393, 2022, https://doi.org/10.28991/ESJ-2022-06-06-010.
and Clustering) Approaches of Big Data Mining in Different Area of Research Journal of Critical Reviews a Review on Machine
Learning (Feature Selection, Classification and Clustering) Approac,” Article in Journal of Critical Reviews, vol. 7, no. August,
p. 2020, 2020, https://doi.org/10.31838/jcr.07.19.322.
[2] N. Rtayli and N. Enneya, “Enhanced credit card fraud detection based on SVM-recursive feature elimination and hyperparameters
optimization,” Journal of Information Security and Applications, vol. 55, no. September, 2020, https://doi.org/
10.1016/j.jisa.2020.102596.
[3] Y. Yin, J. Jang-Jaccard, W. Xu, A. Singh, J. Zhu, F. Sabrina, and J. Kwak, “IGRF-RFE: a hybrid feature selection method for
MLP-based network intrusion detection on UNSW-NB15 dataset,” Journal of Big Data, vol. 10, no. 1, 2023, https://doi.org/10.
1186/s40537-023-00694-8.
[4] H. Jeon and S. Oh, “applied sciences Hybrid-Recursive Feature Elimination for E ffi cient Feature Selection,” Applied Sciences,
pp. 1–8, 2020.
[5] R. Zebari, A. Abdulazeez, D. Zeebaree, D. Zebari, and J. Saeed, “A Comprehensive Review of Dimensionality Reduction
Techniques for Feature Selection and Feature Extraction,” Journal of Applied Science and Technology Trends, vol. 1, no. 2, pp.
56–70, 2020, https://doi.org/10.38094/jastt1224.
[6] M. Ahsan, R. Gomes, M. M. Chowdhury, and K. E. Nygard, “Enhancing Machine Learning Prediction in Cybersecurity Using
Dynamic Feature Selector,” Journal of Cybersecurity and Privacy, vol. 1, no. 1, pp. 199–218, 2021, https://doi.org/10.3390/
jcp1010011.
[7] P. Saraswat, “Supervised Machine Learning Algorithm: A Review of Classification Techniques,” Smart Innovation, Systems
and Technologies, vol. 273, pp. 477–482, 2022, https://doi.org/10.1007/978-3-030-92905-3 58.
[8] J. Cervantes, F. Garcia-Lamont, L. Rodr´ıguez-Mazahua, and A. Lopez, “A comprehensive survey on support vector machine
classification: Applications, challenges and trends,” Neurocomputing, vol. 408, no. xxxx, pp. 189–215, 2020, https://doi.org/10.
1016/j.neucom.2019.10.118.
[9] B. R. Murlidhar, H. Nguyen, J. Rostami, X. N. Bui, D. J. Armaghani, P. Ragam, and E. T. Mohamad, “Prediction of flyrock
distance induced by mine blasting using a novel Harris Hawks optimization-based multi-layer perceptron neural network,”
Journal of Rock Mechanics and Geotechnical Engineering, vol. 13, no. 6, pp. 1413–1427, 2021, https://doi.org/10.1016/j.
jrmge.2021.08.005.
[10] S. M. Cherian and K. Rajitha, “Random forest and support vector machine classifiers for coastal wetland characterization using
the combination of features derived from optical data and synthetic aperture radar dataset,” Journal of Water and Climate
Change, vol. 15, no. 1, pp. 29–49, 2024, https://doi.org/10.2166/wcc.2023.238.
[11] E. Elgeldawi, A. Sayed, A. R. Galal, and A. M. Zaki, “Hyperparameter tuning for machine learning algorithms used for arabic
sentiment analysis,” Informatics, vol. 8, no. 4, pp. 1–21, 2021, https://doi.org/10.3390/informatics8040079.
[12] E. S. El-Kenawy and M. Eid, “Hybrid gray wolf and particle swarm optimization for feature selection,” International Journal
of Innovative Computing, Information and Control, vol. 16, no. 3, pp. 831–844, 2020, https://doi.org/10.24507/ijicic.16.03.831.
[13] M.Weiel, M. G¨otz, A. Klein, D. Coquelin, R. Floca, and A. Schug, “biomolecular simulation parameters with flexible objective
functions,” Nature Machine Intelligence, vol. 3, no. August, pp. 727–734, 2021, https://doi.org/10.1038/s42256-021-00366-3.
[14] E. M. Senan, M. H. Al-Adhaileh, F.W. Alsaade, T. H. Aldhyani, A. A. Alqarni, N. Alsharif, M. I. Uddin, A. H. Alahmadi, M. E.
Jadhav, and M. Y. Alzahrani, “Diagnosis of Chronic Kidney Disease Using Effective Classification Algorithms and Recursive
Feature Elimination Techniques,” Journal of Healthcare Engineering, vol. 2021, 2021, https://doi.org/10.1155/2021/1004767.
[15] J. Xia, L. Sun, S. Xu, Q. Xiang, J. Zhao, W. Xiong, Y. Xu, and S. Chu, “A model using support vector machines recursive
feature elimination (Svm-rfe) algorithm to classify whether copd patients have been continuously managed according to gold
guidelines,” International Journal of COPD, vol. 15, pp. 2779–2786, 2020, https://doi.org/10.2147/COPD.S271237.
[16] S. Zhao and Z. Zhao, “A Comparative Study of Landslide Susceptibility Mapping Using SVM and PSO-SVM Models Based
on Grid and Slope Units,” Mathematical Problems in Engineering, vol. 2021, 2021, https://doi.org/10.1155/2021/8854606.
[17] M. F. Akbar, M. I. Mazdadi, H. Saragih, and F. Abadi, “Implementation of Information Gain Ratio and Particle Swarm Optimization
in the Sentiment Analysis Classification of Covid-19 Vaccine Using Support Vector Machine,” vol. 5, no. 4, pp.
261–270, 2023.
[18] J. Wang, X. Wang, X. Li, and J. Yi, “A Hybrid Particle Swarm Optimization Algorithm with Dynamic Adjustment of Inertia
Weight Based on a New Feature Selection Method to Optimize SVM Parameters,” Entropy, vol. 25, no. 3, 2023, https://doi.org/
10.3390/e25030531.
[19] C. Aroniadi and G. N. Beligiannis, “Applying Particle Swarm Optimization Variations to Solve the Transportation Problem
Effectively,” Algorithms, vol. 16, no. 8, 2023, https://doi.org/10.3390/a16080372.
[20] A. Performance, E. Alshdaifat, D. Alshdaifat, A. Alsarhan, F. Hussein, and S. Moh, “The Effect of Preprocessing Techniques ,
Applied to Numeric,” Data, vol. 6, no. 11, 2021.
[21] M. Ramkumar Prabhu, R. Sivaraman, N. Nagabhooshanam, R. Sampath Kumar, and S. Sathish Salunkhe, “Empowering artificial
intelligence-based multi-biometric image sensor for human identification,” Measurement: Sensors, vol. 33, no. April, p.
101082, 2024, https://doi.org/10.1016/j.measen.2024.101082.
[22] U. M. Khaire and R. Dhanalakshmi, “Stability of feature selection algorithm: A review,” Journal of King Saud University -
Computer and Information Sciences, vol. 34, no. 4, pp. 1060–1073, 2022, https://doi.org/10.1016/j.jksuci.2019.06.012.
[23] K. Shah, H. Patel, D. Sanghvi, and M. Shah, “A Comparative Analysis of Logistic Regression , Random Forest and KNN
Models for the Text Classification,” Augmented Human Research, vol. 0, 2020, https://doi.org/10.1007/s41133-020-00032-0.
[24] J. Alcaraz, M. Labb´e, and M. Landete, “Support Vector Machine with feature selection : A multiobjective approach,” vol. 204,
no. September 2021, 2022.
[25] M. Pan, C. Li, R. Gao, Y. Huang, H. You, T. Gu, and F. Qin, “Photovoltaic power forecasting based on a support vector machine
with improved ant colony optimization,” Journal of Cleaner Production, vol. 277, 2020, https://doi.org/10.1016/j.jclepro.2020.
123948.
[26] M. Jain, V. Saihjpal, N. Singh, and S. B. Singh, “An Overview of Variants and Advancements of PSO Algorithm,” Applied
Sciences (Switzerland), vol. 12, no. 17, pp. 1–21, 2022, https://doi.org/10.3390/app12178392.
[27] H. Reza, R. Zaman, and F. Soleimanian, An improved particle swarm optimization with backtracking search optimization
algorithm for solving continuous optimization problems. Springer London, 2021, no. 0123456789, https://doi.org/10.1007/
s00366-021-01431-6.
[28] N. Van Tinh, “Enhanced Forecasting Accuracy of Fuzzy Time Series Model Based on Combined Fuzzy C-Mean Clustering
with Particle Swam Optimization,” International Journal of Computational Intelligence and Applications, vol. 19, no. 2, 2020,
https://doi.org/10.1142/S1469026820500170.
[29] S. Surono, K. W. Goh, C. W. Onn, A. Nurraihan, N. S. Siregar, A. B. Saeid, and T. T. Wijaya, “Optimization of Markov
Weighted Fuzzy Time Series Forecasting Using Genetic Algorithm (GA) and Particle Swarm Optimization (PSO),” Emerging
Science Journal, vol. 6, no. 6, pp. 1375–1393, 2022, https://doi.org/10.28991/ESJ-2022-06-06-010.
Published
2024-07-01
How to Cite
Sya’idah, I., Surono, S., & Khang Wen, G. (2024). DynamicWeighted Particle Swarm Optimization - Support Vector Machine Optimization in Recursive Feature Elimination Feature Selection. MATRIK : Jurnal Manajemen, Teknik Informatika Dan Rekayasa Komputer, 23(3), 625-638. https://doi.org/https://doi.org/10.30812/matrik.v23i3.3963
Section
Articles
This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.
.png)
