Supplemental Learning Online

I am quite confident that many of you heard of Khan Academy previously.  The "one man university" was created and still maintained by a single individual that teaches various content in small, ten minute portions.  Various other sites in your regular blog reading has touched on it (The Bayesian Heresy and Boing Boing are the first two hits on my feed reader when searching for it).  However, if you do need an introduction, you can check out the site's extensive FAQ section or briefly browse the Wikipedia page on it.  Being intrigued by these alternative education videos, I have been devoting a bit of time every few days to brush up on my calculus – while I do have plenty ofbooks covering the format, an alternative online format complements my own studies.  Khan is obviously not the only one providing online instruction as multiple universities are providing online courses and lectures to both their students and to the mass public – the latter of which provides tuition free enrichment courses via podcasts and webpages.

However, this post is not about Khan Academy, it's merely the introduction.  Freakonomics recently linked to a new study by Figilio, Rush, and Yin with a brief synopsis of their study comparing and contrasting in-person lectures versus online lectures.  As an educator who has taught both online and in person courses, I am quite pleased with the abstract suggesting that there may be some level of job security in what I am doing (as opposed to my courses being distributed via peer-to-peer sites while I wait in unemployment lines):

This paper presents the first experimental evidence on the effects of live versus internet media of instruction. Students in a large introductory microeconomics course at a major research university were randomly assigned to live lectures versus watching these same lectures in an internet setting, where all other factors (e.g., instruction, supplemental materials) were the same. Counter to the conclusions drawn by a recent U.S. Department of Education meta-analysis of non-experimental analyses of internet instruction in higher education, we find modest evidence that live-only instruction dominates internet instruction. These results are particularly strong for Hispanic students, male students, and lower-achieving students. We also provide suggestions for future experimentation in other settings.

Given the experiment, it is obvious that the results are far from conclusive.  Citing the budget crunch, the authors are looking to see whether or not online lectures can serve as a replacement for brick and mortar style of teaching.  As such, the style of the experiment is appropriate in randomly assigning students to watch the lecture either in person or online.  Everything else about the course remained the same.  For what the study is seeking to accomplish, it does its job (while discussing the various experimental threats they perceive). 

The study is a good read and should be taken as a caution of exclusively replacing the classroom with online time.  However, there are multiple lessons not to take away from the study.  Freakonomics offers it as a counter to the US Department of Education (DoEd) report that concludes online teaching to be effective in place of live instruction.  The Freakonomic's quoted portion of the report argues in full:

Students who took all or part of their class online performed better, on average, than those taking the same course through traditional face-to-face instruction. Learning outcomes for students who engaged in online learning exceeded those of students receiving face-to-face instruction, with an average effect size of +0.24 favoring online conditions.3 The mean difference between online and face-to-face conditions across the 51 contrasts is statistically significant at the p < .01 level.4 Interpretations of this result, however, should take into consideration the fact that online and face-to-face conditions generally differed on multiple dimensions, including the amount of time that learners spent on task. The advantages observed for online learning conditions therefore may be the product of aspects of those treatment conditions other than the instructional delivery medium per se. (Emphasis Original)

Contextualized, the experiment is not a direct counter, but an aside or a contribution to our understanding of various teaching mediums.  The DoEd study seeks to see if internet based learning helps across four dimensions:

1.    How does the effectiveness of online learning compare with that of face-to-face instruction?
2.    Does supplementing face-to-face instruction with online instruction enhance learning?
3.    What practices are associated with more effective online learning?
4.    What conditions influence the effectiveness of online learning? (Emphasis Original)

In effect, the DoEd study seems most apt in answering questions of supplementing learning via online instruction and concludes in the affirmative. It concludes with its own caution of interpretation:

However, several caveats are in order: Despite what appears to be strong support for online learning applications, the studies in this meta-analysis do not demonstrate that online learning is superior as a medium, In many of the studies showing an advantage for online learning, the online and classroom conditions differed in terms of time spent, curriculum and pedagogy. It was the combination of elements in the treatment conditions (which was likely to have included additional learning time and materials as well as additional opportunities for collaboration) that produced the observed learning advantages. At the same time, one should note that online learning is much more conducive to the expansion of learning time than is face-to-face instruction.

As such, this does not seem so objectionable and I would venture that this would be generally right.  To bring this back to Khan University at the start of this post, this can be a question of how the supplemental material is added/treated in a class.  If the supplemental material consists of voluntary extensions, then it will likely benefit those who are motivated to take it up.  Alternatively, if it is intended to be mandatory (such as the primary lecture in a course), then the benefits of personal versus online instruction may be found purely in the users.  Generally, the culture of the users is going to guide how they employ a mandatory resource.  If the users think that a course lecture is partly voluntary, skippable, or should be saved for later cramming, then those skipping lectures in person or online will do it as frequently as it is cheap to do so.  There are little social consequences to avoiding an online lecture
while losing face time with the professor (even in 190 person class) can be seen as rude or coming back to to haunt the student by a vengeful professor.  Consequently, even if we have seemingly randomized and non-distinguishable samples in the econ class, the low-cost behavior of skipping an online lecture should be more prevalent (and result in lower grades…if the lecture material is determinant of grades in the course).

Also, I wonder if social networks for studying material are better fostered in a live setting than they are in an online course.  While the motivated few are likely to find those networks offline or online, the mutual shared agony of an introductory economics course may provide bonding in the live setting that may be absent in the atomized confinement of one's dorm room.