In this paper we discuss the development and implementation of a questionnaire that measures preparedness to teach secondary school science with information and communication technologies (ICT). The questionnaire was designed for the purpose of evaluating the effectiveness of instruction in a science education unit in a Graduate Diploma teacher education course. Rasch analysis of the pre- and post-unit responses indicated domains of expertise for which students perceived they had improved their knowledge during the unit, and domains for which they perceived they were less knowledgeable after the unit than before it. We discuss students' responses in relation to the unit, and report the technical decisions that we made as part of the analysis. The questionnaire could be adapted easily to suit preservice teacher education in disciplines other than science. (Contains 1 table and 3 figures.)