Optimasi Hyperparameter Random Forest menggunakan Bayesian Optimization Untuk Prediksi Kelulusan Mahasiswa
DOI:
https://doi.org/10.56127/jts.v5i2.2946Keywords:
Bayesian Optimization; Early Warning System; Machine Learning; Random Forest; Student Graduation PredictionAbstract
The Timely student graduation is one of the key indicators used to evaluate higher education performance. Early identification of students at risk of delayed graduation enables universities to implement appropriate academic interventions and improve student success rates. This study aims to develop a student graduation prediction model using the Random Forest algorithm and to optimize its hyperparameters through Bayesian Optimization. A synthetic dataset consisting of 750 student records was employed, incorporating demographic, socioeconomic, and academic variables. The research methodology included data preprocessing, one-hot encoding, stratified data splitting with an 80:20 ratio for training and testing sets, baseline Random Forest model development, Bayesian hyperparameter optimization using the Tree-structured Parzen Estimator (TPE) with 20 optimization trials, and model evaluation using accuracy, precision, recall, F1-score, receiver operating characteristic area under the curve (ROC-AUC), and average precision. Experimental results showed that the baseline Random Forest achieved an accuracy of 70.67%, precision of 75.00%, recall of 71.43%, F1-score of 73.17%, and ROC-AUC of 82.74%. Bayesian Optimization identified the optimal hyperparameter configuration consisting of 350 trees, a maximum tree depth of six, a minimum split size of seven samples, and a minimum leaf size of four samples. Although the optimized model produced identical accuracy and F1-score values, it improved the ROC-AUC to 83.66% and the average precision to 87.68%, indicating better probability discrimination between the two classes. Feature importance analysis revealed that the number of failed courses, first-semester GPA, cumulative GPA after two semesters, attendance rate, and repeated courses were the most influential predictors. These findings demonstrate that Bayesian Optimization enhances the probabilistic ranking capability of Random Forest and can support the development of an early warning system for identifying students at risk of delayed graduation. However, further validation using real-world academic data is required before practical implementation.
References
Batool, S., Rashid, J., Nisar, M. W., Kim, J., Kwon, H. Y., & Hussain, A. (2023). Educational data mining to predict students’ academic performance: A survey study. Education and Information Technologies, 28, 905–971. https://doi.org/10.1007/s10639-022-11152-y
Beaulac, C., & Rosenthal, J. S. (2019). Predicting university students’ academic success and major using random forests. Research in Higher Education, 60(7), 1048–1064. https://doi.org/10.1007/s11162-019-09546-y
Bergstra, J., & Bengio, Y. (2012). Random search for hyper-parameter optimization. Journal of Machine Learning Research, 13, 281–305.
Breiman, L. (2001). Random forests. Machine Learning, 45(1), 5–32. https://doi.org/10.1023/A:1010933404324
Chaka, C. (2022). Educational data mining, student academic performance prediction, prediction methods, algorithms and tools: An overview of reviews. Journal of e-Learning and Knowledge Society, 18(2), 1–14. https://doi.org/10.20368/1971-8829/1135578
Feurer, M., Klein, A., Eggensperger, K., Springenberg, J. T., Blum, M., & Hutter, F. (2015). Efficient and robust automated machine learning. In C. Cortes, N. Lawrence, D. Lee, M. Sugiyama, & R. Garnett (Eds.), Advances in neural information processing systems (Vol. 28, pp. 2962–2970). Curran Associates.
Hardman, J., Paucar-Caceres, A., & Fielding, A. (2013). Predicting students’ progression in higher education by using the random forest algorithm. Systems Research and Behavioral Science, 30(2), 194–203. https://doi.org/10.1002/sres.2130
Hellas, A., Ihantola, P., Petersen, A., Ajanovski, V. V., Gutica, M., Hynninen, T., Knutas, A., Leinonen, J., Messom, C., & Liao, S. N. (2018). Predicting academic performance: A systematic literature review. In Proceedings companion of the 23rd annual ACM conference on innovation and technology in computer science education (pp. 175–199). Association for Computing Machinery. https://doi.org/10.1145/3293881.329578_predicting
Hutter, F., Hoos, H. H., & Leyton-Brown, K. (2011). Sequential model-based optimization for general algorithm configuration. In C. A. C. Coello (Ed.), Learning and intelligent optimization (pp. 507–523). Springer. https://doi.org/10.1007/978-3-642-25566-3_40
Lindauer, M., Eggensperger, K., Feurer, M., Biedenkapp, A., Deng, D., Benjamins, C., Ruhkopf, T., Sass, R., & Hutter, F. (2022). SMAC3: A versatile Bayesian optimization package for hyperparameter optimization. Journal of Machine Learning Research, 23(54), 1–9.
Nachouki, M., Mohamed, E. A., Mehdi, R., & Abou Naaj, M. (2023). Student course grade prediction using the random forest algorithm: Analysis of predictors’ importance. Trends in Neuroscience and Education, 33, 100214. https://doi.org/10.1016/j.tine.2023.100214
Okoye, K., Nganji, J. T., & Hosseini, S. (2024). A machine learning model and ensemble algorithm for predicting students’ retention and graduation status in education. Decision Analytics Journal, 10, 100383. https://doi.org/10.1016/j.dajour.2023.100383
Probst, P., Wright, M. N., & Boulesteix, A. L. (2019). Hyperparameters and tuning strategies for random forest. WIREs Data Mining and Knowledge Discovery, 9(3), Article e1301. https://doi.org/10.1002/widm.1301
Realinho, V., Machado, J., Baptista, L., & Martins, M. V. (2022). Predicting student dropout and academic success. Data, 7(11), Article 146. https://doi.org/10.3390/data7110146
Shahriari, B., Swersky, K., Wang, Z., Adams, R. P., & de Freitas, N. (2016). Taking the human out of the loop: A review of Bayesian optimization. Proceedings of the IEEE, 104(1), 148–175. https://doi.org/10.1109/JPROC.2015.2494218
Snoek, J., Larochelle, H., & Adams, R. P. (2012). Practical Bayesian optimization of machine learning algorithms. In Advances in neural information processing systems (Vol. 25, pp. 2951–2959). Curran Associates.
Villar, A., & de Santana, Á. L. (2024). Supervised machine learning algorithms for predicting student dropout and academic success: A comparative study. Discover Artificial Intelligence, 4, Article 2. https://doi.org/10.1007/s44163-023-00079-z
Yağcı, M. (2022). Educational data mining: Prediction of students’ academic performance using machine learning algorithms. Smart Learning Environments, 9, Article 11. https://doi.org/10.1186/s40561-022-00192-z
