Task difficulty is a critical issue affecting test developers. Controlling or balancing the item difficulty of an assessment improves its validity and discrimination. Test developers construct tests from the cognitive perspective, by making the test constructing process more scientific and efficient; thus, the scores obtained more precisely represent the proficiency level. In this paper, a framework of cognitive factors related to English listening comprehension (LC) tests is proposed. Data used were derived from the After School Alternative Program (ASAP) English test item pool. A total of 150 items were analyzed, and item difficulty was concurrently calibrated according to three-parameter-logistic item response theory from the responses of 1,459 fifth- to seventh-grade participants. Components affecting item difficulty were proposed and discussed with regard to the cognitive perspective. The 53.5% of variance in item difficulty of LC can be explained by the cognitive predicting model. This result is expected to make the task constructing procedure more organized and to ensure the task is of the desired difficulty, thus enabling the intended ability to be effectively measured. In addition, the ASAP English test aims to assist low-achieving students in remedial instruction; the cognitive factors and study results provide a reference for developing teaching materials of English LC remedial instruction and for constructing LC test items.