Assessing Conversational AI Usability: A System Usability Scale Evaluation of Dola AI
DOI:
https://doi.org/10.56127/juit.v5i2.1324Keywords:
conversational artificial intelligence, Dola AI, System Usability Scale, usability, user experienceAbstract
The rapid development of conversational artificial intelligence has increased the need to evaluate not only system functionality but also the quality of user interaction. Dola AI is an AI-based conversational assistant designed to support users through natural-language interaction; however, widespread use does not necessarily indicate satisfactory usability. Objective: This study aims to evaluate the perceived usability of Dola AI using the System Usability Scale (SUS) and to determine its overall level of user acceptance. Method: A quantitative descriptive approach was employed involving 50 Dola AI users. Data were collected through an online questionnaire containing the ten standard SUS items measured using a five-point Likert scale. The instrument was evaluated through validity and reliability testing, followed by standardized SUS score calculation and interpretation using adjective-rating and acceptability classifications. Findings: All questionnaire items met the validity criterion, while the instrument obtained a Cronbach’s alpha of 0.6827, indicating adequate internal consistency based on the criterion adopted in the study. The overall SUS score was 69.6, placing Dola AI in the “OK” adjective category and the “Marginal” acceptability range. These results indicate that Dola AI provides an adequate level of perceived usability but has not yet reached a clearly high level of user acceptance. Implications: The findings provide practical evidence for improving Dola AI, particularly in terms of interaction consistency, perceived complexity, learnability, and user confidence. The study also demonstrates the applicability of SUS as a concise standardized instrument for evaluating conversational AI interfaces. Originality: This study provides a standardized empirical usability assessment of Dola AI, for which quantitative usability evidence remains limited. The findings establish an initial usability benchmark that can support future comparative studies and further evaluation of conversational AI applications using broader samples and complementary usability measures.
References
Adamopoulou, E., & Moussiades, L. (2020). An overview of chatbot technology. In I. Maglogiannis, L. Iliadis, & E. Pimenidis (Eds.), Artificial intelligence applications and innovations (pp. 373–383). Springer. https://doi.org/10.1007/978-3-030-49186-4_31
Bangor, A., Kortum, P. T., & Miller, J. T. (2008). An empirical evaluation of the System Usability Scale. International Journal of Human-Computer Interaction, 24(6), 574–594. https://doi.org/10.1080/10447310802205776
Bangor, A., Kortum, P., & Miller, J. (2009). Determining what individual SUS scores mean: Adding an adjective rating scale. Journal of Usability Studies, 4(3), 114–123.
Borsci, S., Malizia, A., Schmettow, M., van der Velde, F., Tariverdiyeva, G., Balaji, D., & Chamberlain, A. (2022). The Chatbot Usability Scale: The design and pilot of a usability scale for interaction with AI-based conversational agents. Personal and Ubiquitous Computing, 26, 95–119. https://doi.org/10.1007/s00779-021-01582-9
Borsci, S., Schmettow, M., Malizia, A., Chamberlain, A., & van der Velde, F. (2023). A confirmatory factorial analysis of the Chatbot Usability Scale: A multilanguage validation. Personal and Ubiquitous Computing, 27, 317–330. https://doi.org/10.1007/s00779-022-01690-0
Brooke, J. (1996). SUS: A “quick and dirty” usability scale. In P. W. Jordan, B. Thomas, B. A. Weerdmeester, & I. L. McClelland (Eds.), Usability evaluation in industry (pp. 189–194). Taylor & Francis.
Clark, L., Doyle, P., Garaialde, D., Gilmartin, E., Schlögl, S., Edlund, J., Aylett, M., Cabral, J., Munteanu, C., Edwards, J., & Cowan, B. R. (2019). The state of speech in HCI: Trends, themes and challenges. Interacting with Computers, 31(4), 349–371. https://doi.org/10.1093/iwc/iwz016
Feine, J., Gnewuch, U., Morana, S., & Maedche, A. (2019). A taxonomy of social cues for conversational agents. International Journal of Human-Computer Studies, 132, 138–161. https://doi.org/10.1016/j.ijhcs.2019.07.009
Følstad, A., & Brandtzaeg, P. B. (2020). Users’ experiences with chatbots: Findings from a questionnaire study. Quality and User Experience, 5, Article 3. https://doi.org/10.1007/s41233-020-00033-2
Følstad, A., Nordheim, C. B., & Bjørkli, C. A. (2018). What makes users trust a chatbot for customer service? An exploratory interview study. In S. S. Bodrunova (Ed.), Internet Science (pp. 194–208). Springer. https://doi.org/10.1007/978-3-030-01437-7_16
International Organization for Standardization. (2018). Ergonomics of human-system interaction—Part 11: Usability: Definitions and concepts (ISO 9241-11:2018). International Organization for Standardization.
Jain, M., Kumar, P., Kota, R., & Patel, S. N. (2018). Evaluating and informing the design of chatbots. In Proceedings of the 2018 Designing Interactive Systems Conference (pp. 895–906). Association for Computing Machinery. https://doi.org/10.1145/3196709.3196735
Lewis, J. R. (2018). The System Usability Scale: Past, present, and future. International Journal of Human-Computer Interaction, 34(7), 577–590. https://doi.org/10.1080/10447318.2018.1455307
Lewis, J. R., & Sauro, J. (2018). Item benchmarks for the System Usability Scale. Journal of Usability Studies, 13(3), 158–167.
Nicolescu, L., & Tudorache, M. T. (2022). Human-computer interaction in customer service: The experience with AI chatbots A systematic literature review. Electronics, 11(10), 1579. https://doi.org/10.3390/electronics11101579
Peres, S. C., Pham, T., & Phillips, R. (2013). Validation of the System Usability Scale (SUS). Proceedings of the Human Factors and Ergonomics Society Annual Meeting, 57(1), 192–196. https://doi.org/10.1177/1541931213571043
Rapp, A., Curti, L., & Boldi, A. (2021). The human side of human-chatbot interaction: A systematic literature review of ten years of research on text-based chatbots. International Journal of Human-Computer Studies, 151, 102630. https://doi.org/10.1016/j.ijhcs.2021.102630
Setiawan, A., Kurniawan, E., & others. (2023). Procedia Computer Science.
Downloads
Published
How to Cite
Issue
Section
Citation Check
License
Copyright (c) 2026 Arimbi Kurniasari, Navizha AS Velisya
This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.
