The aim of this study was to investigate the effectiveness of scenario-based multiple-choice questions to assess students' science process skills. To achieve this objective, a test with 32 scenario-based multiple-choice questions evaluating students' skills in formulating hypotheses and identifying variables was prepared and administered to 370 high school freshmen. The questions were involved experiments with two different parts. Both parts of the experiments had the same dependent variable, and in each part the effect of a different manipulated variable on the dependent variable was examined. Therefore, the variables changed roles within the same experiment. In evaluating the test, questions about the first part of the experiments were coded A, and questions about the second part of the experiments were coded B. When the students' scores from the code A and code B items were compared, statistically significant differences were found. Analysis of the data revealed that some students were affected by the different roles played by the variables in the different parts of the experiment.