Background: Governments in low-income countries and donors have invested the equivalent of hundreds of millions of dollars in enhancing reading outcomes. Although reliable measures exist to assess emergent literacy skills in international contexts, there is little consensus on the assessment of reading comprehension. Methods: Using data from 5,389 Kenyan children attending low-cost private schools, we compared the reading comprehension measure from the Early Grade Reading Assessment to two alternatives. We examined relationships among children's outcome scores and the degree to which children's scores on the three measures drew on the same skills. Results: Scores on the three assessments were strongly correlated, and the assessments appeared to draw on the same abilities. The patterns observed in the exploratory factor analyses held for subgroups, including fluent and nonfluent readers and across testing languages. Conclusion: The alternatives did not provide more information on reading comprehension, and we therefore recommend continued use of the Early Grade Reading Assessment standard measure.