Health literacy, defined as an individual's capacity to process health information in order to make appropriate health decisions, is the focus of increasing attention in medical fields due to growing awareness that suboptimal health literacy is associated with poorer health outcomes. To explore this issue, a number of instruments, reported to have high internal consistency and strong correlations with general literacy tests, have been developed. However, their validity as measures of the target construct is seldom explored using multiple sources of evidence. The current study, involving collaboration between health professionals and language specialists, set out to assess the validity of the Rapid Estimate of Adult Literacy in Medicine (REALM), which describes itself as a "reading recognition" test that measures ability to pronounce common medical and lay terms. Drawing on a sample of 310 respondents, including both native and non-native speakers of English, investigations were undertaken to probe the REALM's validity as a measure of understanding the selected terms and to consider associations between scores on this widely used test and those derived from other recognized health literacy tests. Results suggest that the REALM is underrepresenting the health literacy construct and that the test may also be biased against non-native speakers of English. The study points to an expanded role for language testers, working in collaboration with experts from medical disciplines, in developing and evaluating health literacy tools. (Contains 9 tables and 1 figure.)