Prior research has shown that classroom-based evaluation tools are unable to adequately capture the unique aspects of online courses. To address such challenges, a mid-sized Canadian university with a broad and diverse programme of online offerings adapted a questionnaire designed to assess constructivist-compatible online teaching practices. This paper presents the results of a two-phase pilot project to evaluate the appropriateness of the modified tool. The methods section describes the questionnaire, Bangert's Student Evaluation of Online Teaching Effectiveness, the participants in the project, and the techniques used to analyse the data. The results are presented in terms of a validity framework used to guide the evaluation process. Finally, the paper offers critical insights into the key findings from the pilot project and the future of teaching effectiveness evaluation at the institution.