An Overview of the book, >>The No Significant Difference Phenomenon<< by Thomas L. Russell

An Overview of the book, >>The No Significant Difference Phenomenon<< by Thomas L. Russell

In the early 1990s, “Instructional Telecommunications”—as directed by Thomas L. Russell at North Carolina State University (Raleigh)—would mostly have meant delivering course content by video to users as far afield as Japan. Computer-aided instruction, if any, would have been text-based, a high-tech version of the correspondence course.

Correspondence or extension courses had been offered by universities since the nineteenth century. (American variants of distance learning included one devoted to the education of women of all classes, and the educational camp format of the Chautauqua movement.) Radio instruction, popular in the early 20th century, failed at the time as a medium for accredited coursework (although it later hit its stride internationally, along with the podcast). Televised instruction, dating back to 1934, came into its own with a wave of televised programs in the 1960s. With the introduction of e-mail, and with the newly established World Wide Web offering the possibility of delivering multimedia, we come to the landscape occupied by Russell’s department around 1992.

In this environment, Russell—along with the colleagues he kept up with via online “newsgroups”—would often field questions from faculty and administrators unfamiliar with using technology for distance learning: How effective is long-distance education? Do distance learners suffer compared to students in classrooms? And as interest grew in newly-developing delivery media, Russell would be asked why he wasn’t adopting the Next Big Thing in mediated instruction.

Attempts to demonstrate through research that distance learners were not being short-changed dated back to at least 1928. In fact, this kind of research constituted a field of inquiry, Media Comparison Studies, abbreviated MCS. In some 200 studies published by the early 90’s, Russell saw an overwhelming consistency in the results. But for a handful of mutually contradictory studies, the research returned the same result over and over: no significant difference in outcomes between students taught in a classroom and taught at a distance (using “significant”, of course, in the statistical sense). Moreover, this finding was consistent independent of the medium of instruction. Not only were distant students learning as well as on-campus students, but much-vaunted “interactivity” in distance education displayed no advantage, and video instruction with trendy and expensive “high production values” was no more effective than more simply produced “point-and-shoot” videos (or for that matter, correspondence courses).

Russell judged the consistency of the results so striking that it constituted a phenomenon. When in 1992 he published an annotated MCS research bibliography online for his colleagues, he titled it accordingly: The No Significant Difference Phenomenon.

Russell’s contribution was not a “meta-analysis” that tries to draw specific conclusions from a selective aggregate of data recorded and methods used, but an annotated bibliography of the existing research that met basic requirements for methodological soundness. As simple as its approach may have been, Russell’s bibliography became a phenomenon of its own. After four online editions, the fifth edition of The No Significant Difference Phenomenon: A Comparative Research Annotated Bibliography on Technology for Distance Education—by that time listing 355 studies—was published in book form in 1999. It won the Instructional Telecommunications Council’s award for Outstanding Publication in the Area of Distance Learning for that year. Though relatively rare in public libraries, you would be hard pressed to find a university library that does not have a copy—particularly if the university has an education or instructional technology department. (A 2001 reprint by a specialty publisher is still available.) As of this writing, Google Scholar shows some 1400 primary citations of Russell’s work, along with at least scores of secondary citations.

Russell did not claim the studies showed that long-distance, technology-based instruction—or classroom instruction—was equally effective for all students, but that individual variations would tend to cancel each other out. His hope was that better-designed research would succeed the studies on his list, with less vested interest in defending technology-assisted learning or touting new technologies, and with a sharper focus on course design and on student needs and responses. Russell’s bibliography-cum-manifesto certainly did not impede the production of further Media Comparison Studies. But it is safe to say that most subsequent research has been influenced by Russell’s admonition to stop asking the wrong questions. Woe betide the researcher whose proposal for a new MCS does not take the No Significant Difference phenomenon into account! Even Russell’s would-be critics seem to come to similar inferences.

Despite the sharp tone and strong language sometimes used by Russell, he would probably be the last to suggest the baseline findings he reported are dogma rather than the “phenomenon” he describes. He continues to edit the No Significant Difference (NSD) list online, and has long invited submissions of additional research, including studies that do find significant differences and attribute the difference to technology. The NSD site lists some 65 studies (most since 1990) purporting to show better outcomes with technology-delivered content; four studies since 2000 claiming superiority for classroom instruction; and some half-dozen studies as far back as 1984 with “mixed results”. Meanwhile, new NSD studies continue to be produced or old ones “discovered”, so that their aggregate number now approaches 500. The rise of Significant Difference studies may reflect a shift in the question behind Media Comparison Studies in the 1980s from “Can we demonstrate that distance delivery does not harm learning?” to “Can we improve learning outcomes through the use of technology?” Russell’s book briefly comments on the growth of Significant Difference results favoring technological delivery:


[Though] there is nothing inherent in the technologies that elicits improvements in learning…difference in outcomes can be made more positive by adapting the content to the technology. [In] going through the process of redesigning a course to adapt the content to the technology, it can be improved.

The US Department of Education’s 2009 meta-analysis of research on the effectiveness of online learning reported that online instruction has an edge over classroom instruction and that both modes of delivery are less effective alone than a “blended” mode using elements of both. At the same time, it offers several Russell-like caveats about fashionable but apparently ineffective pedagogical practices, and goes so far as to state, “the studies in this meta-analysis do not demonstrate that online learning is superior as a medium”—pointing out that students’ sense of control and willingness to spend more time with the material were a mixed function of, respectively, course design and factors extraneous to the technology itself.

Russell’s motivation for publishing his bibliography was both pragmatic and somewhat polemical. In a 1997 article for EduCom Review, he repeats his own assessment of the “folly” of further research asking simply whether the technology used to deliver instruction improves learning: “Clearly, it does not; however, it does not diminish it either.” He goes on to repeat as still pressing the questions he’d asked five years earlier, a mix of professional soul-searching and suggestions for further areas of research:

  1. Why are empirical research results ignored to the detriment of constituencies?
  2. Why do professional educators embrace high-cost technologies when low-cost technologies work as well?
  3. Why do administration and faculty—despite research results—perceive that distance education technologies, especially those without interaction, are inferior?
  4. Why does interactivity achieve no better results in learning, when individual students and teachers believe that it does?
  5. How can technology-based distance student dropout rates be improved?

As the added italics point out, Russell’s questions show not only impatience with extravagance, orthodoxy, and educational faddism, but a commitment to policy and action based on empirical evidence and, above all, on the best outcome for students.