Administrators of distance learning are faced with the challenge of encouraging reluctant faculty to consider online teaching. Is it possible that supporting faculty to conduct modest research studies is such an avenue? Currently, many research studies compare student grades or scores from an online course to a more traditional one, ignoring confounding variables, such as individual differences, and the impact of instructional design. Yet despite criticism of the comparison study, it continues to be used. Are faculty using these studies to produce replicable results or are they using this design as a way to explore web-based learning and to prove to themselves that it is an acceptable innovation? If so, these studies can be seen as more properly the province of qualitative research that chronicles a personal journey focusing on what the researcher has learned rather than producing valid and replicable research results.