This paper examines how different test methods may tap different aspects of second language knowledge. It employs multiple-choice (MC) and constructed response (CR) items which yield distinct or convergent information in the computer delivered testing of English in its presentation of this factor. In order to examine the effects of test method, a CFA approach to a multitrait-multimethod design was adopted to examine the convergent as well as discriminant validity of the two skill area traits and the two test methods. After comparing the hypothesized model with a nested series of more restrictive models using X[superscript 2] difference tests (Byrne, 2006; Widaman, 1985), based on the information from the path coefficients, and also using principal factor analyses, it was found that: 1) the skills are factorially separable, yet, highly correlated, 2) MC and CR methods are separable, and very weakly correlated, 3) there is thus a method effect, and 4) the MC method may not be suitable in sampling the unique ability characteristics of reading and listening skills.