Online and on-demand tests are increasingly used in assessment. Although the main focus has been cheating and test security (e.g., Selwyn, 2008) the cross-setting equivalence of scores as a function of contrasting test conditions is also an issue that warrants attention. In this study, the impact of environmental and cognitive distractions, as well as personality, on performance on a computerized mathematical problem-solving test was examined by comparing results in a standardized, proctored (school) setting with a non-standardized, unproctored (home) setting. Science and engineering college students (N = 315) were randomly assigned to one of these two test settings and completed a mathematical reasoning test. As hypothesized, model-based analysis demonstrated that greater environmental and cognitive distractions were reported in the unproctored (home) setting, while performance levels were higher in the proctored (school) setting. Furthermore, the process by which these variables impact performance was found to differ between settings. Namely, a statistical comparison of multiple path models supported the hypothesis that cognitive distractions are primarily responsible for the impact of both test anxiety and self-regulation on test performance, although this effect is stronger in non-standardized setting. Implications of these findings for Web, home-based educational courses, and assessments are discussed. (Contains 1 figure and 2 tables.)