Abstract
Jill Watson is an LLM-powered conversational AI partner integrated with instructor-provided courseware, offering learners contextually relevant and immediately applicable support. This study examines learner-generated questions as part of organic interactions with Jill embedded within classroom Learning Management System and investigates whether Jill empowers learners to ask higher-order questions. Leveraging Bloom’s Taxonomy to assess question complexity, we collected over 5500 student questions from classroom deployments across three academic semesters and two educational settings. Student questions were classified using a fine-tuned BERT model and regression models were used to analyze the trends of complexity of the questions over time. Our results reveal a significant proportion of higher-order questions being asked in our classrooms, exceeding typical educational distributions. We also found a statistically significant increase in higher-order questioning with sustained interaction with Jill. These findings demonstrate that Jill empowers learners to engage in critical questioning, thereby enhancing their educational experience by promoting depth, relevance, and application of course concepts. Further research is recommended with larger and more diverse samples to generalize these findings.
Can an AI Partner Empower Learners to Ask Critical Questions
