Grounded in the Community of Inquiry framework, our study uses large language models (LLMs) to detect and summarize cognitive presence - the depth of student reasoning and reflection. The team, comprised of Director of Research in Education Innovation Jeonghyun Lee, Postdoctoral Researcher Gayane Grigoryan, Graduate Research Assistant Rushil Desai, Digital Learning Data Analyst Adrian J. Gallard, and Research Scientist Meryem Yilmaz Soylu, analyzed over 1,500 posts from an introductory MOOC and a graduate-level computer science course. Using LLaMA-based models, they developed classification pipelines to identify cognitive phases, including triggering events, exploration, integration, and resolution.

Our best-performing models achieved an agreement rate of over 92% with human coders, demonstrating strong accuracy and reliability. In addition to classification, we used LLMs to summarize cognitive trajectories across discussion threads, revealing patterns in student thinking. Both single-agent and multi-agent architectures showed promise in enhancing model robustness and interpretability.

This work contributes to the growing field of AI-enhanced learning analytics, providing instructors with practical tools to gain insights and offer feedback in large-scale digital learning environments. Building on this achievement, the team plans to conduct further experiments to enhance our AI models, aiming for a more nuanced understanding and detection of students’ critical thinking and reasoning in online discussion forums. The goal is to share insights that help educators foster richer and more meaningful conversations in digital classrooms. 

DEMOcon, previously known as LWMOOCS, is a renowned international forum for collaboration, networking, learning, and sharing the latest advancements in MOOCs and digital education.

Pictured: Meryem Yilmaz Soylu, Gayane Grigoryan, Adrian J. Gallard, and Jeonghyun Lee.