When a psychologist announces a new research finding, it is often based on a rejected null hypothesis. However, if that hypothesis is true, the claim is a false alarm. Many students mistakenly believe that the probability of committing a false alarm equals alpha, the criterion for statistical significance, which is typically set at 5%. Instructors should take specific steps to dispel this belief because it leads students to misinterpret statistical test results and it reinforces the more general misconception that results can be interpreted in isolation, without reference to theory or prior research. In the present study, students worked with a web app that shows how the false alarm rate is a function of the prior probability of an effect, statistical power, and alpha. Quiz scores suggest the activity helps correct the misconception, which can improve how students conduct and interpret research.