Concerns about the need for assessing multidialectal listening skills for global contexts are becoming increasingly prevalent. However, the inclusion of multiple accents on listening assessments may threaten test fairness because it is not practical to include every accent that may be encountered in the language use domain on these tests. Given this dilemma, this study aimed to determine the extent to which accent strength and familiarity affect comprehension and to provide a defensible direction for assessing multidialectal listening comprehension. A strength of accent scale was developed, and one US, four Australian, and four British English speakers of English were selected based on a judgment of their strength of accent. Next, TOEFL test takers (N = 21,726) were randomly assigned to listen to a common lecture given by one of the nine selected speakers, and respond to six comprehension items and a survey designed to assess their familiarity with various accents. The results suggest that strength of accent and familiarity do affect listening comprehension, and these factors affect comprehension even with quite light accents.