This paper includes two experiments, one investigating complexity factors in stoichiometry word problems, and the other identifying students' problem-solving protocols by using eye-tracking technology. The word problems used in this study had five different complexity factors, which were randomly assigned by a Web-based tool that we developed. The logistic regression analysis in the first experiment of this study showed that the ability of a student to achieve a correct answer was dependent on three complexity factors: "number format," "unit," and "chemical equation." This was followed by an eye-tracking experiment, which reaffirmed that eye fixation durations were different between students at different levels of success when they solved chemistry word problems. The online tool in this study provides a general instrument to design chemistry word problems with various complexities, as well as a method to quantitatively assess problem difficulty and students' cognitive load. This can be applied in constructing curricula and examinations with desired difficulties in chemistry courses.