Home   >   CSC-OpenAccess Library   >    Manuscript Information
An Assessment of the Usability of ChatGPT
Johnathon Hall, Damian Schofield
Pages - 1 - 14     |    Revised - 31-12-2024     |    Published - 01-02-2025
Volume - 13   Issue - 1    |    Publication Date - February 2025  Table of Contents
MORE INFORMATION
KEYWORDS
Artificial Intelligence, Large Language Models, ChatGPT, Human-Computer Interaction, Usability, Ethical Implications, Biological Science.
ABSTRACT
The prevalence of Artificial Intelligence (AI), in particular Large Language Models (LLMs) in multiple areas of society is rapidly growing. This surge in popularity has attracted users of all age groups, leading to a substantial increase in the number of individuals interacting with AI tools. This research aims to examine the way both current and new users engage with ChatGPT (a generative AI chatbot developed by OpenAI and launched in 2022) specifically for academic purposes, and evaluate the effectiveness of this engagement. The project seeks to scrutinize the accuracy of the results produced by ChatGPT, as well as the functionality of the interface and prompt generation within ChatGPT. Additionally, concerns regarding the ethical implications of employing an AI agent for academic research and writing along with accessibility and availability are examined. The study involves college students specializing in the field of biological science as well as their use of ChatGPT to develop research reports. This study finds that despite advances in ChatGPT, users struggle with creating effective inputs due to user interface challenges. It calls for improved LLM interfaces and user education while emphasizing the need for equitable access and ethical considerations by developers.
Afkarin, M. Y., & Asmara, C. H. (2024). Investigating the implementation of ChatGPT in English language education. Journey: Journal of English Language and Pedagogy, 7(1), 57-66.
Alkaissi, H., & McFarlane, S. I. (2023). Artificial hallucinations in ChatGPT: implications in scientific writing. Cureus, 15(2).
Baek, T. H., & Kim, M. (2023). Is ChatGPT scary good? How user motivations affect creepiness and trust in generative artificial intelligence. Telematics and Informatics, 83, 102030.
Chaka, C. (2023). Detecting AI content in responses generated by ChatGPT, YouChat, and Chatsonic: The case of five AI content detection tools. Journal of Applied Learning and Teaching, 6(2).
Chang, Y., Wang, X., Wang, J., Wu, Y., Yang, L., Zhu, K., ... & Xie, X. (2024). A survey on evaluation of large language models. ACM Transactions on Intelligent Systems and Technology, 15(3), 1-45.
Currie, G. M. (2023). Academic integrity and artificial intelligence: is ChatGPT hype, hero or heresy?. Seminars in Nuclear Medicine, 53(5), 719-730.
Dhaini, M., Poelman, W., & Erdogan, E. (2023). Detecting chatgpt: A survey of the state of detecting chatgpt-generated text. arXiv preprint arXiv:2309.07689.
Goldstein, J. A., Sastry, G., Musser, M., DiResta, R., Gentzel, M., & Sedova, K. (2023). Generative language models and automated influence operations: Emerging threats and potential mitigations. arXiv preprint arXiv:2301.04246.
Guo, Z., Jin, R., Liu, C., Huang, Y., Shi, D., Yu, L., ... & Xiong, D. (2023). Evaluating large language models: A comprehensive survey. arXiv preprint arXiv:2310.19736.
Hanna, Robert (2023). How and why ChatGPT failed the Turing Test. Unpublished MS. Available online atURL=https://www.academia.edu/94870578/How_and_Why_ChatGPT_ Failed_The_Turing_Test_January_2023_version.
Hanses, S., & Wang, J. (2022, April). How do users interact with AI features in the workplace? Understanding the AI feature user journey in enterprise. In proceedings of 2022 CHI Conference on Human Factors in Computing Systems (pp. 1-7).
Kalla, D., Smith, N., Samaah, F., & Kuraku, S. (2023). Study and analysis of chat GPT and its impact on different fields of study. International journal of innovative science and research technology, 8(3).
Kantorovitch, J., Niskanen, I., Malins, J., Maciver, F.& Didaskalou A. (2017). Supporting The Initial Stages of The Product Design Process: Towards KnowledgeAwareness And Inspiration, International Journal of Human Computer Interaction, 8(1), 8-22.
Katz, D. M., Bommarito, M. J., Gao, S., & Arredondo, P. (2024). Gpt-4 passes the bar exam. Philosophical Transactions of the Royal Society A, 382(2270), 20230254.
Kuang, E., Li, M., Fan, M., & Shinohara, K. (2024, May). Enhancing UX Evaluation Through Collaboration with Conversational AI Assistants: Effects of Proactive Dialogue and Timing. In Proceedings of the CHI Conference on Human Factors in Computing Systems (pp. 1-16).
Kung, T. H., Cheatham, M., Medenilla, A., Sillos, C., De Leon, L., Elepaño, C., Madriaga, M., Aggabao, R., Diaz-Candido, G., Maningo, J.,& Tseng, V. (2023). Performance of ChatGPT on USMLE: potential for AI-assisted medical education using large language models. PLoS Digital Health, 2(2), e0000198.
Laskar, M. T. R., Alqahtani, S., Bari, M. S., Rahman, M., Khan, M. A. M., Khan, H., ... & Huang, J. (2024). A Systematic Survey and Critical Review on Evaluating Large Language Models: Challenges, Limitations, and Recommendations. Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, 13785–13816
Lechien, J. R., Maniaci, A., Gengler, I., Hans, S., Chiesa-Estomba, C. M., & Vaira, L. A. (2024). Validity and reliability of an instrument evaluating the performance of intelligent chatbot: the Artificial Intelligence Performance Instrument (AIPI). European Archives of Oto-Rhino-Laryngology, 281(4), 2063-2079.
Lee, B. C., & Chung, J. (2024). An empirical investigation of the impact of ChatGPT on creativity. Nature Human Behaviour, 8(10), 1906-1914.
Lee, S. A., Welch, J., Wallace, R. J., Cross, D., & Loffi, J. M. (2024). ChatGPT in the Classroom: A Practical Guide for Educators. Journal of Aviation/Aerospace Education & Research, 33(3), 1.
Liu, Y., Xu, Y., & Song, R. (2023). Transforming User Experience (UX) through Artificial Intelligence (AI) in interactive media design, In proceedings of the IConnect 2023: ES3Web of Conferences (pp. 1-9).
Lu, Y., Yang, Y., Zhao, Q., Zhang, C., & Li, T. J. J. (2024). AI Assistance for UX: A Literature Review Through Human-Centered AI. arXiv preprint arXiv:2402.06089.
Martinelli, S., Lopes, L., & Zaina, L. (2024). UX research practices related to long-term UX: A systematic literature review. Information and Software Technology, 107431.
Mortazavi, E., Doyon-Poulin, P., Imbeau, D., Taraghi, M., & Robert, J. M. (2024). Exploring the landscape of UX subjective evaluation tools and UX dimensions: A Systematic Literature Review (2010–2021). Interacting with Computers, iwae017.
Noever, D., & Ciolino, M. (2022). The turing deception. arXiv preprint arXiv:2212.06721.
OpenAI. (2023) GPT-4 Technical Report. 2023, ArXiv https://doi.org/10.48550/arXiv.2303.08774
Pant, A., Hoda, R., Spiegler, S. V., Tantithamthavorn, C., & Turhan, B. (2024). Ethics in the age of AI: an analysis of ai practitioners’ awareness and challenges. ACM Transactions on Software Engineering and Methodology, 33(3), 1-35.
Plevris, V., Papazafeiropoulos, G., & Jiménez Rios, A. (2023). Chatbots put to the test in math and logic problems: a comparison and assessment of ChatGPT-3.5, ChatGPT-4, and Google bard. AI, 4(4), 949-969.
Rahimi, F., & Abadi, A. T. B. (2023). ChatGPT and publication ethics. Archives of Medical Research, 54(3), 272-274.
Rudolph, J., Tan, S., & Tan, S. (2023). ChatGPT: Bullshit spewer or the end of traditional assessments in higher education?. Journal of applied learning and teaching, 6(1), 342-363.
Sahib, T. M., Alyasiri, O. M., Younis, H. A., Akhtom, D., Hayder, I. M., Salisu, S., & Besse, D. M. (2023). A comparison between ChatGPT-3.5 and ChatGPT-4.0 as a tool for paraphrasing English Paragraphs. In proceedings ofInternational Applied Social Sciences (C-IASOS-2023) Congress (pp. 471-480).
Shah, A., Iftikhar, S. & Chaudhry, N. (2022). Reviewing Assistive Human-Robot Experiences for Inclusive Human-Robot Interaction, International Journal of Recent Trends in Human Computer Interaction (IJHCI), 11(1), 1-18.
Skrabut, S. (2023). 80 Ways to use ChatGPT in the classroom: Using AI to enhance teaching and learning, Skrabut.
Stige, Å., Zamani, E. D., Mikalef, P., & Zhu, Y. (2024). Artificial intelligence (AI) for user experience (UX) design: a systematic literature review and future research agenda. Information Technology & People, 37(6), 2324-2352.
Urban, M., Dechterenko, F., Lukavský, J., Hrabalová, V., Svacha, F., Brom, C., & Urban, K. (2024). ChatGPT improves creative problem-solving performance in university students: An experimental study. Computers & Education, 215, 105031.
Vetter, M. A., Lucia, B., Jiang, J., & Othman, M. (2024). Towards a framework for local Interrogation of AI ethics: A case study on text generators, academic integrity, and composing with ChatGPT. Computers and Composition, 71, 102831.
Xu, W., Dainoff, M., Ge, L.& Gao, Z. (2023) Transitioning toHuman Interaction with AI Systems: New Challenges and Opportunities for HCI Professionals toEnable Human-Centered AI, International Journal of Human–Computer Interaction, 39(3), 494-518.
Xu, Y., Liu, Y., Xu, H., & Tan, H. (2024). AI-driven UX/UI design: Empirical research and applications in FinTech. International Journal of Innovative Research in Computer Science & Technology, 12(4), 99-109.
Yang B., Wei L., Pu, Z.(2020). Measuring and Improving User Experience Through Artificial Intelligence-Aided Design. Frontiers in Psychology, 11.
Yang, Q., Steinfeld, A., Rosé, C., & Zimmerman, J. (2020, April). Re-examining whether, why, and how human-AI interaction is uniquely difficult to design. In proceedings of the 2020 CHI Conference on Human Factors in Computing Systems (pp. 1-13).
Yilmaz, F. G. K., Yilmaz, R., & Ceylan, M. (2024). Generative artificial intelligence acceptance scale: A validity and reliability study. International Journal of Human–Computer Interaction, 40(24), 8703-8715.
Zhuo, T. Y., Huang, Y., Chen, C., & Xing, Z. (2023). Exploring ai ethics of chatgpt: A diagnostic analysis. arXiv preprint arXiv:2301.12867, 10(4).
Zhuo, Y., Li, Z., & Li, H. (2023). Evaluating Large Language Models: A Comprehensive Survey. arXiv preprint arXiv:2310.19736.
Mr. Johnathon Hall
Department of Computer Science, State University of New York, Oswego, New York, 13126Department of Computer Science, State University of New York, Oswego, New York, 13126 - United States of America
Professor Damian Schofield
Department of Computer Science, State University of New York, Oswego, New York, 13126 - United States of America
damian.schofield@oswego.edu


CREATE AUTHOR ACCOUNT
 
LAUNCH YOUR SPECIAL ISSUE
View all special issues >>
 
PUBLICATION VIDEOS