Making inferences about population differences based on samples of data, that is, performing intuitive analysis of variance (IANOVA), is common in everyday life. However, the intuitive reasoning of individuals when making such inferences (even following statistics instruction), often differs from the normative logic of formal statistics. The present study examined the reasoning used by several cohorts of first year statistics students when performing IANOVA. In general, participants perceived datasets representing larger but less reliable group differences as stronger evidence of a population effect than datasets representing smaller yet more reliable differences, across various data formats (Experiment 1) and datasets (Experiment 2). Qualitative results revealed several distinct patterns of reasoning between participants which was associated with performance. Implications for instruction are discussed.