March 17, 2020
Online Pedagogy: Lessons learned from teaching an online course in Georgia Tech’s OMCSCS program
Ashok K. Goel (goel@cc.gatech.edu) Design & Intelligence Laboratory, School of Interactive Computing, Georgia Institute of Technology
Introduction
The spread of COVID-19 has led to an unprecedented move towards online education across the country and around the world. It is too early to predict the long-term impact of this extraordinary move: it may be temporary and much of education may soon revert back to the old normal of in-person classes, or it may lead to a large-scale shift towards online learning. Even if the move is only temporary, it may lead to a new normal of blended learning in which online educational materials, home works and assessments support in-person learning. In any case, this raises fundamental questions for online pedagogy: How do we develop a successful online course? How do we prepare high quality educational materials for online classes?
In 2014, my Georgia Tech colleague David Joyner and I faced exactly these questions when we developed an online course on Knowledge-Based Artificial Intelligence (KBAI) as part of Georgia Tech’s Online Masters of Science in Computer Science program (OMSCS; http://www.omscs.gatech.edu/). We identified about 150 KBAI concepts, methods and skills we wanted students to learn, developed a set of 26 video lessons, and designed a suite of learning assessments including home works, design and programming projects, and take home examinations (Goel & Joyner 2016, 2017). In Fall 2014, we offered the online class to about 200 students in the OMSCS program. In Fall 2015, I transformed the pedagogy I had been using in the in-person KBAI class for graduate and undergraduate residential students for more than a decade into blended learning (Goel 2019). In Spring 2016, my Design & Intelligence research laboratory developed a virtual teaching assistant called Jill Watson for automatically answering routine questions on the discussion forum of the online KBAI class (Goel & Polepeddi 2017). We estimate that more than 6000 students have taken the online and blended versions of the KBAI class since Fall 2014 and more than 150 (human) teaching assistants have helped with the teaching and learning in the class.
Assessing the quality of learning in any class is a complex matter, whether the class is online, blended or in-person. Three types of data indicate that the quality of learning in the online KBAI class is comparable to that in the in-person class for residential students (Goel & Joyner 2016, 2017; Goel 2019). First, surveys of online students from Fall 2014 through Fall 2019 report the same kind and degree of satisfaction with the online KBAI class as do the residential students with the in-person KBAI class during the same period. Second, the completion ratio in the online KBAI classes during this period has been comparable to that in the in-person KBAI classes, which is atypical for online classes. Third, the performance of the online KBAI students on the same set of learning assessments has been comparable to that of the residential students in the in-person KBAI classes. To control for the difference in the student demographics between the online and residential students, in Fall 2018 and Fall 2019, we repeated the above quasi-experimental studies in online and in-person sections of the KBAI class offered only to residential students, and found similar results.
Design Principles for Online Classes
In Goel & Joyner (2016), we analyze the design principles for developing online classes in detail. Table 1 summarizes the 10 design principles underlying the online and blended KBAI classes. For example, the first principle suggests that the course instructor may want to explicitly the learning goals, outcomes, strategies and assessments before developing the online class. The eighth principle indicates that design of an online class is an iterative process, with each iteration based on feedback and reflection on the preceding iteration. Thus, it is important to collect and analyze feedback and leave time for deliberation and reflection.
Table 1: Design Principles for Online Classes (adapted from Goel & Joyner 2016)
|
|
|
|
|
|
|
|
|
|
Design Principles for Video Lessons
The 26 video lessons for the online and blended versions of the KBAI class embed about 150 exercises, one for each concept in the concept inventory, as well as about 100 tutors that provided adaptive feedback on many of the exercises (https://www.udacity.com/course/knowledge-based-ai-cognitive-systems–ud409). My colleague Chaohua Ou with Georgia Tech’s Center for Teaching and Learning has analyzed the design of the videos, and student responses to them over several semesters, in detail (Ou, Joyner & Goel 2019). Table 2 summarizes the seven principles for designing video lessons for online classes derived from the analysis. For example, the seventh principle suggests the use of prepared visuals rather than drawing them at run time.
Table 2: Design Principles for Video Lessons (adapted from Ou, Joyner & Goel 2019)
|
|
|
|
|
|
|
Reflections
It is important to note three qualifications here. First, developing a successful online class requires expertise not only in the subject of the course, but also in information technology and learning science. It also requires a strong team, and significant financial and technological support. We were very fortunate to have all these assets in developing the online KBAI class in 2014. My research laboratory conducts research into KBAI and thus we had the expertise in the subject; we are computer scientists and thus we had some expertise in information technology as well; and Joyner and I already were developing expertise in learning science as part of his Ph.D. work. In addition, Georgia Tech generously provided financial and technological support for developing the video lessons for the online KBAI class.
Second, if a teacher’s goal is simply to convert the educational materials prepared for an in-person class to an online forum due to the spread of COVID-19 and the consequent need for social distancing, then we expect the design principles in Table 1 should be of immediate value. The principles in both Tables 1 and 2 are likely to add more value if in the longer term a teacher wants to develop a new online course from the start.
Third, learning in general is situated in the external world and thus is context dependent. The effectiveness and efficiency of learning depends not only on the individual learner, teacher, and educational materials, but also on the physical, technological, social and cultural contexts of learning. While we found the design principles enumerated above useful in developing the online KBAI class, we expect that their operationalization will vary across different learning contexts.
References
A. Goel. Preliminary Evidence for the Benefits of Online and Blended Learning. In A. Madden, L. Margulieux, R, Kadel & A, Goel (editors), Blended Learning in Practice: A Guide for Practitioners and Researchers, MIT Press. April 2019.
A. Goel & D. Joyner. An Experiment in Teaching Artificial Intelligence Online. International Journal for Scholarship of Technology-Enhanced Learning, 1(1): 1-27, 2016.
A. Goel & D. Joyner. Using AI to Teach AI: Lessons from an Online AI Class. AI Magazine, 38(2): 48-59, Summer 2017.
A. Goel & L. Polepeddi. Jill Watson, A Virtual Teaching Assistant for Online Education. In C. Dede, J. Richards & B. Saxberg (editors), Education at Scale: Engineering Online Teaching and Learning, NY: Routledge, 2018.
C. Ou, D. Joyner, & A. Goel. Designing and Developing Video Lessons for Online Learning: A Seven-Principle Model. Online Learning Journal, 23(2): 82-104, June 2019.
Author Biography: Ashok Goel is a Professor of Human-Centered computing in the School of Interactive Computing at Georgia Institute of Technology. He is also the Chief Scientist with Georgia Tech’s Center for 21st Century Universities. In 2014, he co-developed an online course on Knowledge-Based AI; in 2016, his research laboratory developed Jill Watson, a virtual teaching assistant for answering questions in online class discussion forums; and in 2019, he co-edited a volume on Blended Learning published by MIT Press. In 2017, Ashok received Georgia Tech’s Class of 1934 Outstanding Innovative Use of Educational Technology Award; in 2019, he received AAAI’s Outstanding AI Educator Award; and in 2020, he received the University System of Georgia’s Faculty Hall of Fame Award for Scholarship of Teaching and Learning.
