COMPARISON OF PRE-TRAINED BERT-BASED TRANSFORMER MODELS FOR REGIONAL LANGUAGE TEXT SENTIMENT ANALYSIS IN INDONESIA
DOI:
https://doi.org/10.56127/ijst.v3i3.1739Keywords:
Sentiment Analysis, BERT, Indonesian Regional Languages, IndoBERT, NusaBERT ArticleAbstract
This study compared the performance of eight pre-trained BERT-based models for sentiment analysis across ten regional languages in Indonesia. The objective was to identify the most effective model for analyzing sentiment in low-resource Indonesian languages, given the increasing need for automated sentiment analysis tools. The study utilized the NusaX dataset and evaluated the performance of IndoBERT (IndoNLU), IndoBERT (IndoLEM), Multilingual BERT, and NusaBERT, each in both base and large variants. Model performance was assessed using the F1-score metric. The results indicated that models pre-trained on Indonesian data, specifically IndoBERT (IndoNLU) and NusaBERT, generally outperformed the multilingual BERT and IndoBERT (IndoLEM) models. IndoBERT-large (IndoNLU) achieved the highest overall F1-score of 0.9353. Performance varied across the different regional languages. Javanese, Minangkabau, and Banjar consistently showed high F1 scores, while Batak Toba proved more challenging for all models. Notably, NusaBERT-base underperformed compared to IndoBERT-base (IndoNLU) across all languages, despite being retrained on Indonesian regional languages. This research provides valuable insights into the suitability of different pre-trained BERT models for sentiment analysis in Indonesian regional languages.
References
Y. Wu, Z. Jin, C. Shi, P. Liang, and T. Zhan, “Research on the Application of Deep Learning-based BERT Model in Sentiment Analysis,” 2024. doi: https://doi.org/10.48550/arXiv.2403.08217.
A. Halim and A. Safuwan, “Analisis Sentimen Opini Warganet Twitter Terhadap Tes Screening Genose Pendeteksi Virus Covid-19 Menggunakan Metode Na"ive Bayes Berbasis Particle Swarm Optimization,” Jurnal Informatika Teknologi dan Sains (Jinteks), vol. 5, no. 1, pp. 170–178, 2023, doi: https://doi.org/10.51401/jinteks.v5i1.2229.
Z. Li, R. Li, and G. Jin, “Sentiment Analysis of Danmaku Videos Based on Naïve Bayes and Sentiment Dictionary,” IEEE Access, vol. 8, pp. 75073–75084, 2020, doi: https://doi.org/10.1109/ACCESS.2020.2986582.
S. Styawati, A. Nurkholis, A. A. Aldino, S. Samsugi, E. Suryati, and R. P. Cahyono, “Sentiment Analysis on Online Transportation Reviews Using Word2Vec Text Embedding Model Feature Extraction and Support Vector Machine (SVM) Algorithm,” in 2021 International Seminar on Machine Learning, Optimization, and Data Science (ISMODE), 2022, pp. 163–167. doi: https://doi.org/10.1109/ISMODE53584.2022.9742906.
M. Aufar, R. Andreswari, and D. Pramesti, “Sentiment Analysis on Youtube Social Media Using Decision Tree and Random Forest Algorithm: A Case Study,” in 2020 International Conference on Data Science and Its Applications (ICoDSA), 2020, pp. 1–7. doi: https://doi.org/10.1109/ICoDSA50139.2020.9213078.
A. Thakkar, D. Mungra, A. Agrawal, and K. Chaudhari, “Improving the Performance of Sentiment Analysis Using Enhanced Preprocessing Technique and Artificial Neural Network,” IEEE Trans Affect Comput, vol. 13, no. 4, pp. 1771–1782, 2022, doi: https://doi.org/10.1109/TAFFC.2022.3206891.
M. R. Raza, W. Hussain, and J. M. Merigó, “Cloud Sentiment Accuracy Comparison using RNN, LSTM and GRU,” in 2021 Innovations in Intelligent Systems and Applications Conference (ASYU), 2021, pp. 1–5. doi: https://doi.org/10.1109/ASYU52992.2021.9599044.
A. Vaswani et al., “Attention is All you Need,” in Advances in Neural Information Processing Systems, I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, Eds., Curran Associates, Inc., 2017. doi: https://doi.org/10.48550/arXiv.1706.03762.
J. Devlin, “Bert: Pre-training of deep bidirectional transformers for language understanding,” arXiv preprint arXiv:1810.04805, 2018, doi: https://doi.org/10.48550/arXiv.1810.04805.
T. B. Brown et al., “Language Models are Few-Shot Learners,” 2020. doi: https://doi.org/10.48550/arXiv.2005.14165.
S. Zein, Language Policy in Superdiverse Indonesia. New York : Routledge, 2020. | Series: Routledge studies in sociolinguistics: Routledge, 2020. doi: https://doi.org/10.4324/9780429019739.
A. F. Aji et al., “One Country, 700+ Languages: NLP Challenges for Underrepresented Languages and Dialects in Indonesia,” 2022. doi: https://doi.org/10.48550/arXiv.2203.13357.
H. Murfi, Syamsyuriani, T. Gowandi, G. Ardaneswari, and S. Nurrohmah, “BERT-based combination of convolutional and recurrent neural network for indonesian sentiment analysis,” Appl Soft Comput, vol. 151, p. 111112, 2024, doi: https://doi.org/10.1016/j.asoc.2023.111112.
H. Imaduddin, F. Y. A’la, and Y. S. Nugroho, “Sentiment analysis in Indonesian healthcare applications using IndoBERT approach,” International Journal of Advanced Computer Science and Applications, vol. 14, no. 8, pp. 113–117, 2023, doi: https://dx.doi.org/10.14569/IJACSA.2023.0140813.
P. Subarkah, P. Arsi, D. I. S. Saputra, A. Aminuddin, Berlilana, and N. Hermanto, “Indonesian Police in the Twitterverse: A Sentiment Analysis Perspectives,” in 2023 IEEE 7th International Conference on Information Technology, Information Systems and Electrical Engineering (ICITISEE), 2023, pp. 76–81. doi: https://doi.org/10.1109/ICITISEE58992.2023.10405357.
B. Wilie et al., “IndoNLU: Benchmark and Resources for Evaluating Indonesian Natural Language Understanding,” 2020. doi: https://doi.org/10.48550/arXiv.2009.05387.
F. Koto, A. Rahimi, J. H. Lau, and T. Baldwin, “IndoLEM and IndoBERT: A Benchmark Dataset and Pre-trained Language Model for Indonesian NLP,” 2020. doi: https://doi.org/10.48550/arXiv.2011.00677.
W. Wongso, D. S. Setiawan, S. Limcorn, and A. Joyoadikusumo, “NusaBERT: Teaching IndoBERT to be Multilingual and Multicultural,” 2024. doi: https://doi.org/10.48550/arXiv.2403.01817.
K. S. Nugroho, A. Y. Sukmadewa, H. Wuswilahaken DW, F. A. Bachtiar, and N. Yudistira, “Bert fine-tuning for sentiment analysis on indonesian mobile apps reviews,” in Proceedings of the 6th International Conference on Sustainable Information Engineering and Technology, 2021, pp. 258–264. doi: https://doi.org/10.48550/arXiv.2107.06802.
L. Geni, E. Yulianti, and D. I. Sensuse, “Sentiment Analysis of Tweets Before the 2024 Elections in Indonesia Using IndoBERT Language Models,” Jurnal Ilmiah Teknik Elektro Komputer Dan Informatika (JITEKI), vol. 9, no. 3, pp. 746–757, 2023, doi: http://dx.doi.org/10.26555/jiteki.v9i3.26490.
F. Basbeth and D. H. Fudholi, “Klasifikasi Emosi Pada Data Text Bahasa Indonesia Menggunakan Algoritma BERT, RoBERTa, dan Distil-BERT,” JURNAL MEDIA INFORMATIKA BUDIDARMA, vol. 8, no. 2, pp. 1160–1170, 2024, doi: http://dx.doi.org/10.30865/mib.v8i2.7472.
T. Pires, E. Schlinger, and D. Garrette, “How Multilingual is Multilingual BERT?,” in Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, A. Korhonen, D. Traum, and L. Màrquez, Eds., Florence, Italy: Association for Computational Linguistics, Jul. 2019, pp. 4996–5001. doi: https://doi.org/10.18653/v1/P19-1493.
G. I. Winata et al., “NusaX: Multilingual Parallel Sentiment Dataset for 10 Indonesian Local Languages,” in Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, A. Vlachos and I. Augenstein, Eds., Dubrovnik, Croatia: Association for Computational Linguistics, May 2023, pp. 815–834. doi: https://doi.org/10.18653/v1/2023.eacl-main.57.
A. Purwarianti and I. A. P. A. Crisdayanti, “Improving Bi-LSTM Performance for Indonesian Sentiment Analysis Using Paragraph Vector,” in 2019 International Conference of Advanced Informatics: Concepts, Theory and Applications (ICAICTA), 2019, pp. 1–5. doi: https://doi.org/10.1109/ICAICTA.2019.8904199.
G. Grefenstette, “Tokenization,” in Syntactic Wordclass Tagging, H. van Halteren, Ed., Dordrecht: Springer Netherlands, 1999, pp. 117–133. doi: 10.1007/978-94-015-9273-4_9.
Y. Wu et al., “Google’s Neural Machine Translation System: Bridging the Gap between Human and Machine Translation,” 2016. doi: https://doi.org/10.48550/arXiv.1609.08144.
T. Kudo and J. Richardson, “SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing,” in Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, E. Blanco and W. Lu, Eds., Brussels, Belgium: Association for Computational Linguistics, Nov. 2018, pp. 66–71. doi: https://doi.org/10.18653/v1/D18-2012.
Z. Lan, M. Chen, S. Goodman, K. Gimpel, P. Sharma, and R. Soricut, “ALBERT: A Lite BERT for Self-supervised Learning of Language Representations,” 2020. doi: https://doi.org/10.48550/arXiv.1909.11942.
T. Nguyen et al., “CulturaX: A Cleaned, Enormous, and Multilingual Dataset for Large Language Models in 167 Languages,” 2023. doi: https://doi.org/10.48550/arXiv.2309.09400.
N. Team et al., “No Language Left Behind: Scaling Human-Centered Machine Translation,” 2022. doi: https://doi.org/10.48550/arXiv.2207.04672.
L. Yang and A. Shami, “On hyperparameter optimization of machine learning algorithms: Theory and practice,” Neurocomputing, vol. 415, pp. 295–316, 2020, doi: https://doi.org/10.1016/j.neucom.2020.07.061.
I. Loshchilov and F. Hutter, “Decoupled Weight Decay Regularization,” 2019. doi: https://doi.org/10.48550/arXiv.1711.05101.
I. Loshchilov and F. Hutter, “SGDR: Stochastic Gradient Descent with Warm Restarts,” 2017. doi: https://doi.org/10.48550/arXiv.1608.03983.
P. Micikevicius et al., “Mixed Precision Training,” 2018. doi: https://doi.org/10.48550/arXiv.1710.03740.
