Operational Weakness Mapping of Machine Learning–Based IntrusionDetection Systems under Realistic Deployment Scenarios

Abstract

As machine learning-based intrusion detection systems increasingly support information security risk management, prior systematic literature review findings indicate that many studies still emphasize benchmark accuracy while paying limited attention to robustness, interpretability, and operational feasibility. This study aims to map the operational weaknesses of machine learning-based intrusion detection systems under realistic deployment stressors. A directed replication and scenario-based stresstesting approach was applied using four public intrusion detection datasets, namely CICIDS2017, CICIDS2018, UNSW-NB15, and RanSMAP. The data were obtained from public repositories, converted to binary labels, cleaned by removing identifiers and non-numeric attributes, imputed with median values, scaled with MinMax normalization, and split into training and testing subsets. Supervised models, including Random Forest and XGBoost, were compared with unsupervised baselines, including Isolation Forest, LOF/kNN-distance, and DBSCAN, across scenarios covering baseline benchmarking, class imbalance, telemetry degradation, drift, parameter sensitivity, and micro-batch inference. The results show that supervised models achieved near-perfect baseline performance but degraded sharply under minor Gaussian noise, with F1-score dropping to 0.16 for Random Forest and 0.41 for XGBoost. Unsupervised models showed limited detection capability and high sensitivity to parameters. Although micro-batch inference achieved high throughput, alert burden remained a practical concern. These findings demonstrate that benchmark accuracy alone is insufficient for deployment readiness and that IDS evaluation should include robustness, interpretability, and alert-management analysis.