One major threat to validity in high-stakes testing is construct-irrelevant variance. In this study we explored whether the transition from a paper-and-pencil to a computer-based test mode in a high-stakes test in China, the College English Test, has brought about variance irrelevant to the construct being assessed in this test. Analyses of the test scores, text complexity, language errors, and writing processes revealed that computer-based writing elicited cognitive processes among test takers that were similar to those of traditional paper-based writing; moreover, a high level of computer familiarity had a facilitative effect on test-takers' performances. In considering these results we argue that, in the 21st century, the construct of a computer-based language test needs to be reconceptualized by drawing on Chalhoub-Deville's view of local context-bound constructs: computer literacy should be viewed as an important contextual facet interacting with the construct measured in a computer-based language assessment.