This paper examines the use of a randomization-based activity to introduce the ANOVA F-test to students. The two main goals of this activity are to successfully teach students to comprehend ANOVA F-tests and to increase student comprehension of sampling distributions. Four sections of students in an advanced introductory statistics course participated in this study. When the topic of ANOVA was introduced, two sections were randomly assigned to participate in a traditional approach (one for each instructor), while the other two sections participated in a randomization-based method. Students were administered a pre-test and a post-test that asked them to apply hypothesis testing concepts to a new scenario. Students also responded to some basic conceptual questions. This study showed mixed results related to students' understanding of the logic of inference, but the activity utilized shows some promising gains in student understanding of sampling distributions.