Comparison of Machine Learning Methods for Classifying User Satisfaction Opinions of the PeduliLindungi Application

Since the emergence of the Covid-19 virus, the Indonesian government urged people to study, work, and worship or work from home. The social restriction policy has changed people’s behavior which requires physical distance in social interaction. The government developed an application to minimize the spread of Covid-19, namely the PeduliLindungi application. The PeduliLindungi application is a tracking application to prevent the spread of Covid-19. The government’s policy of implementing the PeduliLindungi application during Covid-19 aroused pros and cons from the public. The volume of PeduliLindungi application review data on Google Play was increasing, so manual analysis could not be done. New analytical approaches needed to be carried out, such as sentiment analysis. This research aimed to analyze user reviews of the PeduilLindungi application using classiﬁcation methods, namely Support Vector Machine (SVM), Random Forest, and Na¨ıve Bayes. The methods used were Synthetic Minority Oversampling Technique (SMOTE), Random Forest, SVM, and Na¨ıve Bayes. SMOTE was used to balance user review data on the PeduliLindungi application. After the data had been balanced, classiﬁcation was carried out. The results of this study showed that the Random Forest method with SMOTE got better accuracy than the SVM and Na¨ıve


INTRODUCTION
Covid-19 has been a public concern since its appearance was detected in the city of Wuhan, China, in 2020.The emergence of Covid-19 has caused the order of human life to change, especially social life.Since the beginning of the emergence of the covid-19 virus, the Indonesian government has urged to study, work, and worship from home or work from home.People's habits, such as gathering and shaking hands, must be accustomed to social restrictions.The social restriction policy has changed people's behavior which requires physical distance in the process of social interaction.The government itself developed an application that is used to minimize the spread of covid-19, namely the PeduliLindungi application [1].The PeduliLindungi application is a tracking application to prevent the spread of covid-19 [2].
The features that can be used in the PeduliLindungi application are the location of vaccinations and the community sharing location data for traveling so that the contact history with Covid-19 patients can be known.In addition, the application lets users know when they are in the red zone [3].The policy of implementing the use of the PeduliLindungi application by the government during covid-19 arouses the pros and cons of the public.Public complaints about the use of the PeduliLindungi application are (1) frequent errors, (2) vaccine certificates do not appear, (3) vaccine status does not change, and (4) concerns about data leakage [4].In addition, the volume of PeduliLindungi app review data on Google Play is increasing, so manual analysis cannot be done, so a new analytical approach needs to be taken, such as sentiment analysis [5].Sentiment analysis quickly emerged as an automated process to examine semantic relationships and meanings in reviews.One of the sentiment analysis techniques on social media data that can be used is the machine learning approach [5].The machine learning approach utilizes algorithms to extract and detect sentiment from an opinion on social media data.Therefore, this research proposes a machine learning approach for sentiment analysis of user satisfaction in the PeduliLindungi application, such as satisfied, neutral, and dissatisfied automatically.
Several previous studies have discussed user reviews of the PeduliLindungi application using various methods, such as research conducted by [6] using the Naïve Bayes method for analyzing user reviews of the Protect application with a total of 1179 datasets and the accuracy obtained is 83.3%.Research [7] compared the Naïve Bayes and SVM methods for sentiment analysis of the PeduliLindungi application using a dataset of 4636 instances.Based on the results of his research, the SVM method has higher accuracy than Naïve Bayes, which is 91% for SVM and 90% for Naïve Bayes.Research [8] used SVM, Naïve Bayes, and KNN methods to classify community responses to the PeduliLindungi application using a dataset of 6000 instances.Based on the test results of this research using 80% training data and 20% testing data, the SVM method got the best accuracy compared to the Nave Bayes and KNN methods of 76.5%.Research [9] utilized the Naïve Bayes method for sentiment analysis of the Peduli Lindungi application with a dataset of 1000 instances.Based on the test results of this research using 80% training and 20 testing data, the Naïve Bayes method obtained an accuracy of 73%.Research [10] used the Long Short Term Memory (LSTM) method for user sentiment classification in the PeduliLindungi application with review data from as many as 3000 instances divided into positive and negative reviews.The results of his research obtained an accuracy of 82.44%, a precision of 78.66%, and a recall of 87%.
Research [11] used the Naïve Bayes method to classify user reviews of the Peduli Lindungi application, with 496 instances of review data divided into positive and negative reviews.The study results obtained an accuracy of 85%, precision of 77.7%, and recall of 98%.Research [12] combined the Naïve Bayes method with Particle Swarm Optimization (PSO) feature selection to classify user reviews on the PeduliLindungi application with review data of 1364 instances divided into positive and negative reviews.The results of the combination of Naïve Bayes and PSO methods get an accuracy of 93% and AUC of 97.7%.Research [13] used the Convolutional Neural Network (CNN) method for sentiment analysis of PeduliLindungi application user reviews based on aspects of Visual Experience, Scan, Vaccine Certificate, eHac, Covid-19 Test, Login, Performance, and Security.The data used were 2320 instances, and the results of his research obtained an f1 score of 95.13%.Research [14] used the SVM method for sentiment analysis of user reviews on the PeduliLindungi application with an accuracy of 64%.Research [15] used the C4.5 method to classify the nutritional status of toddlers with an accuracy of 95%.
Research conducted by [16] used SVM and Naïve Bayes methods to analyze online news media application reviews on Google Play with a dataset of 5000 instances with an accuracy of 88% for SVM and 87% for Naïve Bayes.Research conducted by [17] used the SVM method for sentiment analysis of electronic money service customers with a dataset of 3852 instances obtained from Twitter with an accuracy of 98%.Finally, research conducted by [18] used the random forest method to analyze comments on the relocation of the country's capital with a dataset of 1639, and the accuracy obtained was 76%.The difference between this research and previous research is this research handles the problem of unbalanced data on PeduliLindungi application reviews using SMOTE.After the data is balanced, the PeduliLindungi application user reviews are classified using three methods, namely SVM, Random Forest, and Naïve Bayes.The novelty of this research is the combination of the SMOTE method with several machine learning methods such as SVM, Random Forest, and Naïve Bayes for classifying user opinion satisfaction in the PeduliLindungi application.Therefore, this study aims to analyze user reviews of the PeduliLindungi application using classification methods, namely SVM, Random Forest, and Naïve Bayes.Ì 433

RESEARCH METHOD
Figure 1 shows the flow of this research.Based on Figure 1, this research consists of several stages: collecting datasets by scraping user reviews on the PeduliLindungi application on Google Play.Next is labeling user review data on the PeduliLindungi application, divided into satisfied, neutral, and dissatisfied reviews based on the rating given.Rating ratings of more than three are considered satisfied reviews, ratings of three become neutral, and ratings of less than three become dissatisfied.The next stage is the text preprocessing stage.The text preprocessing section processes raw data into quality data by case folding, tokenization, filtering, and stemming.The results of text preprocessing are then carried out text weighting using the TF-IDF method.The next stage is text classification using Nave Bayes, Random Forest, and SVM methods based on the division of training and testing data using 10-fold cross-validation.The last stage is performance testing based on accuracy using the confusion matrix table.he data used are user reviews about the PeduliLindungi application on Google Play obtained by scraping data.However, the results of scraping user review data do not yet contain review categories such as satisfied, neutral, and dissatisfied, so it is necessary to label the review category based on the rating value given by the user.For example, a user score rating ≥ 4 becomes Satisfied, a user score rating ≤ 3 becomes Dissatisfied, and a score rating = 3 becomes Neutral.

Text Preprocessing
Text preprocessing is used to process raw data into quality data by performing case folding, tokenization, filtering, and stemming [19].The commonly used stages in the preprocessing stage are shown in Figure 2. In Figure 2, the preprocessing stage begins with case folding.Case folding is the process of changing capital letters into lowercase letters.In contrast, tokenization is the process of separating sentences into words.Then, filtering is used to remove less essential words or keep important words.For example, common words that usually appear in Indonesian are "yang," "dan," "di," and "from," so they need to be removed.Finally, Stemming is the process of forming base words.

Word Weighting
The data from text preprocessing that is still in the form of words will be converted to numerical form with a word weighting process that aims to get the weight of each word used as a feature.The result of word weighting with TF-IDF is the multiplication of TF and IDF values which will produce the weight of each word using Equation (1).Tf is term frequency, W is TF-IDF weight, and idf is Inverse Document Frequency.

Data Sampling
The data used contains class imbalance, which can affect the performance of the classification method.Therefore, unbalanced data must be handled to improve the classification method's performance.This research employed the SMOTE method to balance the data by adding instances to the minority class based on the nearest neighbor.Figure 3 displays the various stages of the SMOTE method.

Implementation Model
At this stage, the algorithms used in classification will be implemented, namely the SVM, Random Forest, and Naïve Bayes methods.The Naïve Bayes classifier is a classification method based on Bayes' theorem.The Naïve Bayes method uses probability and statistical methods to predict opportunities based on previous experience.The Naive Nayes method's classification process uses Equation (2) [20].P(H|X) represents the probability that the hypothesis is true for the observed sample data X.P(X|H) represents the probability of the sample data X if hypothesis H is true.P(H) represents the probability of hypothesis H, and P(X) represents the probability of the observed sample.The SVM method works by separating two different classes using Equation (3).W is the weight, x is the input data, y is the class, and b is the bias.The Random Forest algorithm is an ensemble learning-based classification method.The advantages of Random Forest are that it has good accuracy results, is robust against outliers and noise, and is faster than bagging and boosting [21].The stages of the Random Forest method are shown in Figure 4.

Performance Testing
At this stage, testing is carried out on the classification method used.This research uses accuracy testing to measure performance on the classification method using Equation (4).Accuracy measurement uses a confusion matrix table that consists of True Positive, True Negative, False Positive, and False Negative.True Positive is the actual positive class predicted correctly.True Negative is the actual negative class predicted correctly, False Positive is the actual negative class, but the positive class is predicted.Finally, false Negative is the actual positive class, but the negative class is predicted.

RESULT AND ANALYSIS
This section describes the research results obtained under the stages in Figure 1.PeduliLindungi application user review data is 4005 instances that do not yet have labels, so labeling needs to be done based on user score ratings.For example, a user score rating ≥ 4 becomes Satisfied, a user score rating ≤ 3 becomes Dissatisfied, and a score rating = 3 becomes Neutral.The results of the data labeling that has been done are shown in Table 1.The data that has been obtained is text-processed to make it more qualified with case folding, tokenization, filtering, and stemming.The processed data can be shown in Table 2.The data processed in the previous stage is still in the form of words, so it needs to be converted to numeric form.Changing words into numeric form is required to obtain the weight of each word so that it can be processed by the classification method.The results of word weighting using the TF-IDF method are shown in Table 3 The data used still contains unbalanced data that affects the performance of the classification method.For example, the number of dissatisfied opinion labels (2999 instances) is more than satisfied opinions (506 instances) and neutral (500 instances), so it needs to balance the opinion label data.Therefore, this research uses the SMOTE method to oversample the data so that the data becomes balanced.The amount of unbalanced and balanced data can be seen in Figure 5.The classification methods used in this research are Naïve Bayes, Random Forest, and SVM.The three methods' performance testing is based on accuracy, precision, and specificity using the confusion matrix table.Based on testing the Naïve Bayes, Random Forest, and SVM methods using 10-fold cross-validation, the results are obtained as a confusion matrix table as in Figure 6 for the Naïve Bayes, Figure 7 for the SVM, and Figure 8 for the Random Forest results.In this section, we will analyze the results obtained by the Random Forest, Naïve Bayes, and SVM methods based on accuracy, as shown in Figure 9. Based on Figure 8, the Random Forest method with SMOTE gets the best accuracy compared to the Naïve Bayes and SVM methods.The accuracy obtained by the Random Forest method with Smote is 96.3%.Additionally, using SMOTE can increase the accuracy of classification methods supported by research [22][23][24][25][26].The increase in accuracy in the classification method used after using SMOTE is that the Random Forest method increased by 16.2%, the SVM method increased by 15.4%, and the Naïve Bayes method increased by 4.7%.A comparison of the research results obtained with the results of previous research is shown in Table 4. Table 4 shows that the proposed method gets better accuracy than some previous research results using the same case study.

Figure 6
shows that without SMOTE, the Naïve Bayes method can correctly classify the Satisfied label in 0 instances, Neutral in 88 instances, and Dissatisfied in 2992.With SMOTE, the Naïve Bayes method can classify the Satisfied label in 2625 instances, Neutral in 2583 instances, and Unsatisfied in 2136.According to Figure 7, the SVM method without SMOTE can correctly classify the Satisfied label in 17 instances, Neutral in 258, and Dissatisfied in 2950.However, with SMOTE, the SVM method can classify the Satisfied label in 2776 instances, Neutral in 2916 instances, and Dissatisfied in 2942.Figure 8 shows that without SMOTE, the Random Forest method can correctly classify the Satisfied label in 11 instances, Neutral in 238 instances, and Ì 437 Dissatisfied in 2958.With SMOTE, however, the Random Forest method can classify the Satisfied label in 2866 instances, Neutral in 2913 instances, and Dissatisfied in 2888 instances.

Figure 5 .Figure 6 .Figure 8 .
Figure 5. Data Distribution of Satisfaction Opinion on the Use of PeduliLindungi Application

Figure 9 .
Figure 9. Accuracy of the Classification Methods Used

Table 1 .
Dataset Labelling Result

Table 2 .
Text Preprocessing Result

Table 3 .
Term Weighting Results Using TF-IDF Method

Table 4 .
Comparison Results of this Research with Previous Research