Can ChatGPT pass Glycobiology?

Abstract

The release of text-generating applications based on interactive Large Language Models (LLMs) in late 2022 triggered an unprecedented and ever-growing interest worldwide. The almost instantaneous success of LLMs stimulated lively discussions in public media and in academic fora alike on the value and potentials of such tools in all areas of knowledge and information acquisition and distribution, but also about the dangers posed by their uncontrolled and indiscriminate use. This conversation is now particularly active in the higher education sector, where LLMs are seen as a potential threat to academic integrity at all levels, from facilitating cheating by students in assignments, to plagiarising academic writing in the case of researchers and administrators. Within this framework, we were interested in testing the boundaries of the LLM ChatGPT (www.openai.com) in areas of our scientific interest and expertise, and in analysing the results from different perspectives, i.e. of a final year BSc student, of a research scientist, and of a lecturer in higher education. To this end, in this paper we present and discuss a systematic evaluation on how ChatGPT addresses progressively complex scientific writing tasks and exam-type questions in Carbohydrate Chemistry and Glycobiology. The results of this project allowed us to gain insight on, 1) the strengths and limitations of the ChatGPT model to provide relevant and (most importantly) correct scientific information, 2) the format(s) and complexity of the query required to obtain the desired output, and 3) strategies to integrate LLMs in teaching and learning.