Solving mathematics story problems requires text comprehension skills. However, previous studies have found few connections between traditional measures of text readability and performance on story problems. We hypothesized that recently developed measures of readability and topic incidence measured by text-mining tools may illuminate associations between text difficulty and problem-solving measures. We used data from 3,216 middle and high school students from 10 schools using the Cognitive Tutor Algebra program; these schools were geographically, socioeconomically, racially, and ethnically diverse. We found that several indicators of the readability and topic of story problems were associated with students' tendency to give correct answers and request hints in Cognitive Tutor. We further examined the individual skill of writing an algebraic expression from a story scenario, and examined students at the lowest performing schools in the sample only, and found additional associations for these subsets. Key readability and topic categories that were related to problem-solving measures included word difficulty, text length, pronoun use, sentence similarity, and topic familiarity. These findings are discussed in the context of models of mathematics story problem solving and previous research on text comprehension.