When understanding or evaluating foreign-accented speech, listeners are affected not only by properties of the speech itself but by their own linguistic backgrounds and their experience with different speech varieties. Given the latter influence, it is not known to what degree a diverse group of listeners might share a response to second language (L2) speech. In this study, listeners from native Cantonese, Japanese, Mandarin, and English backgrounds evaluated the same set of foreign-accented English utterances from native speakers of Cantonese, Japanese, Polish, and Spanish. Regardless of native language background, the listener groups showed moderate to high correlations on intelligibility scores and comprehensibility and accentedness ratings. Although some between-group differences emerged, the groups tended to agree on which of the 48 speakers were the easiest and most difficult to understand; between-group effect sizes were generally small. As in previous studies, the listeners did not consistently exhibit an intelligibility benefit for speech produced in their own accent. These findings support the view that properties of the speech itself are a potent factor in determining how L2 speech is perceived, even when the listeners are from diverse language backgrounds.