In this experimental study, 20 multiple-choice test items from the Massachusetts Grade 5 science test were linguistically simplified, and original and simplified test items were administered to 310 English learners (ELs) and 1,580 non-ELs in four Massachusetts school districts. This study tested the hypothesis that specific linguistic features of test items contributed to construct-irrelevant variance in science test scores of ELs. Simplifications targeted specific linguistic features, to identify those features with the largest impacts on ELs' test performance. Of all the linguistic simplifications used in this study, adding visual representations to answer choices had the largest positive effect on ELs' performance. These findings have significant implications for the design of multiple-choice test items that are fair and valid for ELs.