Students' grasp of the non-mathematical language in a mathematics story problem--such as vocabulary and syntax--may have an important effect on their problem-solving, and this may be particularly true for students with weaker language skills. However, little experimental research has examined which individual language features influence students' performance while solving problems--much research has been correlational or has combined language features together. In the present study, we manipulated six different language features of algebra story problems--number of sentences, pronouns, word concreteness, word hypernymy, consistency of sentences, and problem topic--and examined how systematically varying readability demands impacts student performance. We examined both accuracy and response time measures, using an assignment for learning linear functions in the ASSISTments online problem-solving environment. We found little evidence that individual language features have a considerable effect on mathematics word problem solving performance for a general population of students. However, sentence consistency reduced response time and problems about motion or travel had shorter response times than problems about business or work. In addition, it appears students may benefit or be harmed by language modifications depending on their familiarity with ASSISTments. Implications for the role of language in math word problems are discussed.