Clickers are used to improve student learning, motivation, and engagement. Smartphones can serve as clickers; however, instructional use of smartphones may lead to students multitasking between instructional and alternative media. This study investigates whether students are distracted after instructional use of smartphones in a lecture-based classroom. Outcomes were assessed through both self-reported smartphone use and in-class observations of actual smartphone use. Students were observed covertly for 5 minutes following instructional use of smartphones to determine whether multi-tasking distraction occurred and/or persisted following the instructional use of smartphones. Even though the self-reported data indicate that students disagreed to somewhat disagreed that smartphone use was a distraction, our findings show that 42% of students began to use their smartphones for non-instructional purposes immediately following the instructional episode, and 28% of students persisted in this behavior five minutes after the instructional episode ended. The observations contradicted the students' self-reported survey responses, thus emphasizing the need to critically consider self-reported outcomes related to multi-tasking distraction in the classroom. Policies or practices to limit multitasking distraction due to non-instructional use of smartphones in the classroom should be considered in cases where smartphones are being used for instructional purposes.