We know little about what raters rely on when participating in accentedness perception tasks as their qualitative comments are rarely scrutinised. At the same time, we know that (assumed) social information influences listener behaviour. This study investigates rater attitudes to and stereotypes about speakers of different varieties of English, through exploration of qualitative comments about the speakers' accentedness and origin. In the task 30 native speakers of New Zealand English listened to 24 native and non-native speakers of English and (1) rated them on an accentedness scale, (2) guessed their origin, and (3) commented on the language in the audio clips. Besides the expected mentions of non-target-like segments, intonation, and grammar, the listener responses revealed their engagement with social categories. The listeners were found to employ stereotypes about the speakers' paralinguistic behaviour, vocabulary, topic, etc., in identifying their origin, which highlights the importance of socio-cultural knowledge in linguistic tasks.