It is common to administer measures of autistic traits to those without autism spectrum disorders (ASDs) with, for example, the aim of understanding autistic personality characteristics in non-autistic individuals. Little research has examined the extent to which measures of autistic traits actually measure the same traits in the same way across those with and without an ASD. We addressed this question using a multi-group confirmatory factor invariance analysis of the Autism Quotient Short Form (AQ-S: Hoekstra et al. in "J Autism Dev Disord" 41(5):589-596, 2011) across those with (n = 148) and without (n = 168) ASD. Metric variance (equality of factor loadings), but not scalar invariance (equality of thresholds), held suggesting that the AQ-S measures the same latent traits in both groups, but with a bias in the manner in which trait levels are estimated. We, therefore, argue that the AQ-S can be used to investigate possible causes and consequences of autistic traits in both groups separately, but caution is due when combining or comparing levels of autistic traits across the two groups.