Classroom response systems are widely used in interactive teaching environments as a way to engage students by asking them questions. Previous research on the time taken by students to respond to conceptual questions has yielded insights on how students think and change conceptions. We measure the amount of time students take to respond to in-class, conceptual questions [ConcepTests (CTs)] in two introductory physics courses taught using Peer Instruction and use item response theory to determine the difficulty of the CTs. We examine response time differences between correct and incorrect answers both before and after the peer discussion for CTs of varying difficulty. We also determine the relationship between response time and student performance on a standardized test of incoming physics knowledge, precourse self-efficacy, and gender. Our data reveal three results of interest. First, response time for correct answers is significantly faster than for incorrect answers, both before and after peer discussion, especially for easy CTs. Second, students with greater incoming physics knowledge and higher self-efficacy respond faster in both rounds. Third, there is no gender difference in response rate after controlling for incoming physics knowledge scores, although males register significantly more attempts before committing to a final answer than do female students. These results provide insight into effective CT pacing during Peer Instruction. In particular, in order to maintain a pace that keeps everyone engaged, students should not be given too much time to respond. When around 80% of the answers are in, the ratio of correct to incorrect responses rapidly approaches levels indicating random guessing and instructors should close the poll.