Research in second language speech has often focused on listeners' accent judgment and factors that affect their perception. However, the topic of listeners' application of specific sound categories in their own perceptual judgments has not been widely investigated. The current study explored how listeners from diverse language backgrounds weighed phonetic parameters (i.e., segmental features such as consonants and vowels and suprasegmental features such as word stress and sentence stress) differently when perceiving non-native speakers' accented speech. Two hundred forty listeners, including American, Vietnamese, and Arabic students, rated Vietnamese accented English for intelligibility, comprehensibility, and accentedness. Within this group of participants, 112 raters also provided interview responses to questions related to their perception of accented speech in general. The results suggest that listeners of English perceived degree of accent in fundamentally different ways, depending on factors such as their first language and their English instruction backgrounds. Features identified in this study can be useful both in the listeners' global judgments and in the communicative situations in which second language learners need to function.