Purpose: Determining the valid and fair use of the interview for medical school admissions is contingent upon a demonstration of the reproducibility of interview scores. This study seeks to establish the generalizability of interview scores, first assessing the existing research evidence, and then analyzing data from a non-experimental independent replications research design. Methods: Multivariate and univariate generalizability analyses are conducted using data from a structured interview obtained from a population of medical school applicants over two years. Results: The existing literature does not provide sufficient evidence regarding interview reliability. In this study, interview scores derived from a standardized interview were found to display low to moderate levels of reliability. Interview scores do not appear to possess the level of precision found with other measures commonly used to facilitate admissions decisions. Discussion/Conclusion: Given the results obtained, the fairness of using the interview as a highly influential component of the admission process is called into question. Methods for using interview data in a psychometrically defensible fashion are discussed. Specifically, attention to decision reliability provides guidance on how interview scores can best be integrated into the admissions process.