Undergraduate students were asked to complete gas law questions using a Web-based tool as a first step in our understanding of the role of cognitive load in chemistry word questions and in helping us assess student problem-solving. Each question contained five different complexity factors, which were randomly assigned by the tool so that a different question was created for each attempt. Data were collected in general or preparative chemistry courses at four universities. The results were analyzed using logistic regression. Based on the student responses, the regression showed that the students' ability to achieve a correct answer for their assigned question was dependent on three of the five complexity factors: number format, volume unit, and temperature unit. The creation of this tool provides a platform for developing a testing process and understanding student difficulties related to cognitive skills in chemistry. (Contains 7 tables.)