Comparison of Lexicon-Based Methods and Bidirectional Encoder Representations for Transformers Models in Sentiment Analysis of Government Debt Market Movements

Firda Rachmawati; Ulil Azmi; Rahmania Azwarini

doi:10.30812/ijecsa.v4i1.4832

Firda Rachmawati Institut Teknologi Sepuluh Nopember, Surabaya, Indonesia
Ulil Azmi Institut Teknologi Sepuluh Nopember, Surabaya, Indonesia
Rahmania Azwarini Institut Teknologi Sepuluh Nopember, Surabaya, Indonesia

DOI: https://doi.org/10.30812/ijecsa.v4i1.4832

Keywords: Bidirectional Encoder Representations for Transformers (BERT), Government Bonds, Lexicon-Based, Natural Language Processing, Sentiment Analysis

Abstract

The State Budget of Indonesia (APBN) is the main tool for implementing fiscal policies and serves as a budgeting guideline for development execution in Indonesia. One of the funding sources in budget financing is Debt Financing, which consists of Government Securities (SBN) issuance and Loans. Overall, SUN contributes IDR 5,824.34 trillion, highlighting its significant proportion in debt financing. Understanding public sentiment toward SUN is essential in developing effective government policies. This research conducts sentiment analysis on tweets from the social media X over the past 7.75 years to assess public perception and propose strategic recommendations. The aim of this research is to compare the BERT model and the Lexicon-Based method to determine which achieves the highest accuracy in sentiment analysis. The findings can help the government develop strategies for issuing SUN, especially in improving public involvement and investor trust. This research method is based on a deep learning pre-trained Bidirectional Encoder Representations from Transformers (BERT) model, specifically IndoBERT, with fine-tuning, and a Lexicon-Based approach utilizing the InSet lexicon. The results of this research are as follows: on the overall tweet dataset, the BERT model with optimal hyperparameters outperformed the Lexicon-Based method, achieving an accuracy of 70.28% compared to 55.77%. Similarly, on an annual basis, BERT exhibited higher accuracy than the Lexicon-Based method, except in 2021. Public sentiment on SUN in social media X is categorized as 49% positive, 30% neutral, and 21% negative. These findings indicate a generally favorable perception of SUN but also highlight areas for improvement in public communication. Overall, the BERT model demonstrates superior performance over the Lexicon-Based method. Considering the opportunities available, the government could leverage social media through Key Opinion Leaders and enhance transparency in explaining policies such as Tapera. This approach could maximize public participation in investing in SUN in Indonesia.

References

[1] L. S. Laras, “Efektivitas dan Kontribusi Surat Berharga Negara Terhadap Pembiayaan Defisit Anggaran Pendapatan dan Belanja Negara di Indonesia Tahun 2012 – 2021,” Universitas Lampung, 2024.
[2] S. Thomas, T. Informasi, and I. Shanti Bhuana, “Studi Analisis Metode Analisis Sentimen pada YouTube,” JIFOTECH (Journal of Information Technology), vol. 1, no. 1, Mar. 2021.
[3] D. Musfiroh, U. Khaira, P. E. P. Utomo, and T. Suratno, “Analisis Sentimen terhadap Perkuliahan Daring di Indonesia dari Twitter Dataset Menggunakan InSet Lexicon,” MALCOM: Indonesian Journal of Machine Learning and Computer Science, vol. 1, no. 1, pp. 24–33, Mar. 2021, doi: 10.57152/malcom.v1i1.20.
[4] A. Setiawan, “Analisis Sentimen Masyarakat di Twitter terhadap Kejadian Bom Bunuh Diri Polsek Astana Anyar Menggunakan Algoritma SVM dengan Leksikon Vader dan InSet,” UIN Syarif Hidayatullah, Jakarta, 2023.
[5] A. Faizal, A. Susilo, Y. Irawan, and D. Juardi, “Perbandingan Lexicon Based dan Naïve Bayes Classifier Pada Analisis Sentiment Pengguna Twitter terhadap Gempa Turki,” Journal of Information Technology and Computer Science (INTECOMS), vol. 6, no. 2, 2023.
[6] Z. A. Sriyanti, D. S. Y. Kartika, and A. R. E. Najaf, “Implementasi Model BERT pada Analisis Sentimen Pengguna Twitter Terhadap Aksi Boikot Produk Israel,” Jurnal Informatika dan Teknik Elektro Terapan, vol. 12, no. 3, pp. 2335–2342, Aug. 2024, doi: 10.23960/jitet.v12i3.4743.
[7] C. Sahoo, M. Wankhade, and B. K. Singh, “Sentiment Analysis Using Deep Learning Techniques: A Comprehensive Review,” Int J Multimed Inf Retr, vol. 12, no. 2, Dec. 2023, doi: 10.1007/s13735-023-00308-2.
[8] M. F. Ramadhan and B. Siswoyo, “Mengenal Model BERT dan Implementasinya untuk Analisis Sentimen Ulasan Game,” Prosiding SISFOTEK, vol. 8, no. 1, pp. 395–398, Oct. 2024.
[9] R. Mas, R. W. Panca, K. Atmaja1, and W. Yustanti2, “Analisis Sentimen Customer Review Aplikasi Ruang Guru dengan Metode BERT (Bidirectional Encoder Representations from Transformers),” JEISBI, vol. 02, no. 3, pp. 55–62, 2021.
[10] B. Kurniawan, A. Ari Aldino, and A. Rahman Isnain, “Sentimen Analisis terhadap Kebijakan Penyelenggara Sistem Elektronik (PSE) Menggunakan Algoritma Bidirectional Encoder Representations from Transformers (BERT),” Jurnal Teknologi dan Sistem Informasi, vol. 3, no. 4, pp. 98–106, Dec. 2022, [Online]. Available: http://jim.teknokrat.ac.id/index.php/JTSI
[11] S. Tabinda Kokab, S. Asghar, and S. Naz, “Transformer-Based Deep Learning Models for the Sentiment Analysis of Social Media Data,” Array, vol. 14, Apr. 2022, doi: 10.1016/j.array.2022.100157.
[12] V. Chandradev, I. Made, A. Dwi Suarjaya, I. Putu, and A. Bayupati, “Analisis Sentimen Review Hotel Menggunakan Metode Deep Learning BERT,” Jurnal Buana Informatika, vol. 14, no. 2, pp. 107–116, 2023, doi: 10.24002/jbi.v14i02.7244.
[13] Ardiansyah, A. S. Widagdo, K. N. Qodri, F. E. N. Saputro, and N. A. Rizky, “Analisis sentimen terhadap pelayanan Kesehatan berdasarkan ulasan Google Maps menggunakan BERT,” Jurnal Fasilkom, vol. 13, no. 2, pp. 326–333, Aug. 2023.
[14] M. Amien, G. Frendi Gunawan, and K. Kunci, “BERT dan Bahasa Indonesia: Studi tentang Efektivitas Model NLP Berbasis Transformer,” ELANG: Journal of Interdisciplinary Research, vol. 1, no. 2, pp. 132–140, Jan. 2024, doi: 10.32664/elang.v1i02.
[15] Sutoyo, “Pengaruh Surat Berharga Negara (SBN) terhadap Pertumbuhan Ekonomi Indonesia,” Jurnal Ilmiah Sains, Teknologi, Ekonomi, Sosial dan Budaya, vol. 6, no. 2, pp. 86–89, May 2022.
[16] B. Juanda and S. Gladiola, “Analisis Keberlanjutan serta Pengaruh Surat Berharga Negara dan Faktor Lainnya terhadap Pertumbuhan Ekonomi di Indonesia,” Indonesian Treasury Review: Jurnal Perbendaharaan, Keuangan Negara dan Kebijakan Publik, vol. 7, no. 1, pp. 239–254, 2022.
[17] S. Arifin and H. Adjie, “Keamanan Surat Utang Negara yang Dijual Secara Lelang Ditinjau dari Asas Akuntabilitas,” Jurnal Magister Hukum Argumentum, vol. 8, no. 2, pp. 69–76, Sep. 2022.
[18] C. R. Ko and H. T. Chang, “LSTM-Based Sentiment Analysis for Stock Price Forecast,” PeerJ Comput Sci, vol. 7, pp. 1–23, Mar. 2021, doi: 10.7717/peerj-cs.408.
[19] K. F. Ariani, T. I. Rahmawati, and D. V. Anggraini, “Peningkatan Literasi Keuangan Masyarakat Pedesaan Guna Mendorong Tingkat Inklusi Keuangan Indonesia Perspektif Hukum Perbankan,” Jurnal Multidisiplin Ilmu Akademik, vol. 1, no. 6, pp. 118–128, Dec. 2024, doi: 10.61722/jmia.v1i6.2874.
[20] A. Hermawansyah and A. R. Pratama, “Analisis Profil dan Karakteristik Pengguna Media Sosial di Indonesia dengan Metode EFA dan MCA,” Techno.COM, vol. 20, no. 1, pp. 69–82, Feb. 2021, doi: 10.33633/TC.V20I1.4289.
[21] D. A. Editya, “Tinjauan atas Efektivitas Penggunaan Key Opinion Leader (KOL) dalam Penjualan Surat Utang Negara Ritel seri SBR011,” Aug. 2022, doi: 10.48550/arXiv.2208.12619.