Mid-semester formative evaluations of college teaching are a promising, low-cost solution to providing online instructors with in-the-moment feedback to improve their online teaching practices. However, existing instruments suffer from issues of validity and bias, and fail to align with evidence-based strategies. This paper presented and psychometrically validated a research-based Mid-Semester Evaluation of College Teaching for Online Instructors (MSECT-O) among 170 undergraduate students in seven online courses. Pilot study results demonstrated that the MSECT-O is a valid and reliable tool for online educators seeking to improve their online teaching and classroom climate.