A Feasibility Study on the Use of Large Language Models in Supporting Adaptive Learning

Abstract

Large Language Models (LLMs) have gained prominence as adaptive learning tools. Although effective in assessing multiple-choice questions (MCQs), their accuracy and feedback validity remain uncertain. This feasibility study examines the accuracy, validity, and scientific substantiation of LLMs’ responses to expert-generated pharmacology MCQs and assesses the automated generation of MCQs to elucidate their potential roles and limitations in adaptive learning.

Fifty MCQs, designed and validated by academic pharmacists, were used to test the accuracy of four LLMs. Each question was classified according to Bloom’s Taxonomy. Direct prompting was applied to generate responses for each question. Responses were analysed for accuracy, validity of rationale, and the existence and relevance of supporting references. Chi-Square test and Fisher-Freeman-Halton Test were used to evaluate quantitative findings.

Among the four LLMs, ChatGPT-4o achieved the highest accuracy (84%), followed by Google Gemini 1.5 (Gemini) (80%), Microsoft Copilot (Copilot) (72%), and Claude Sonnet 3.5 (Claude) (68%). An answer-rationale discordance and a decline in performance with an increased cognitive complexity, stratified through Bloom’s Taxonomy, were noted across the LLMs. Artificial hallucinations were observed in the study. These limitations underline the challenges of using LLMs in complex, evidence-driven disciplines like pharmacology.

LLMs, as an educational tool, may provide value to adaptive learning. However, limitations in logical reasoning, scientific support, and higher-order thinking highlight the need for cautious adoption. Continuous efforts to validate LLMs with larger, more diverse question banks are also necessary to fully investigate their potential in adaptive learning.