ERIC Number: ED503307
Record Type: Non-Journal
Publication Date: 2007-Jun
Abstractor: As Provided
Developing Academic English Language Proficiency Prototypes for 5th Grade Reading: Psychometric and Linguistic Profiles of Tasks. An Extended Executive Summary. CSE Report 720
Bailey, Alison L.; Huang, Becky H.; Shin, Hye Won; Farnsworth, Tim; Butler, Frances A.
National Center for Research on Evaluation, Standards, and Student Testing (CRESST)
Within an evidentiary framework for operationally defining academic English language proficiency (AELP), linguistic analyses of standards, classroom discourse, and textbooks have led to specifications for assessment of AELP. The test development process described here is novel due to the emphasis on using linguistic profiles to inform the creation of test specifications and guide the writing of draft tasks. In this report, we outline the test development process we have adopted and provide the results of studies designed to turn the drafted tasks into illustrative prototypes (i.e., tried out tasks) of AELP for the 5th grade. The tasks use the reading modality; however, they were drafted to measure the academic language construct and not reading comprehension per se. That is, the tasks isolate specific language features (e.g., vocabulary, grammar, language functions) occurring in different content areas (e.g., mathematics, science, and social studies texts). Taken together these features are necessary for reading comprehension in the content areas. Indeed, students will need to control all these features in order to comprehend information presented in their textbooks. By focusing on the individual language features, rather than the subject matter or overall meaning of a text, the AELP tasks are designed to help determine whether a student has sufficient antecedent knowledge of English language features to be able to comprehend the content of a text. The work reported here is the third and final stage of an iterative test development process. In previous National Center for Research on Evaluation, Standards, and Student Testing (CRESST) work, we conducted a series of studies to develop specifications and create tasks of AELP. Specifically, we first specified the construct by synthesizing evidence from linguistic analyses of ELD and content standards, textbooks (mathematics, science, and social studies), and teacher talk in classrooms, resulting in language demand profiles for the 5th grade. After determining task format by frequency of assessment types in textbooks, we then created draft tasks aligned with the language profiles. The goals of the current effort were to take these previously drafted tasks and create prototypes by trying out the tasks for the first time with 224 students from native English and English language learner (ELL) backgrounds. Students across the 4th-6th grades, as well as native-English students, are included in the studies because native speakers and adjacent grades provide critical information about the targeted language abilities of mainstream students at the 5th grade level. Phase 1 (n= 96) involved various tryouts of 101 draft tasks to estimate duration of administration, clarity of directions, whole-class administration procedures, and an opportunity to administer verbal protocols to provide further information about task accessibility and characteristics. Phase 2, the pilot stage, involved administration of 40 retained tasks (35 of which were modified as a result of Phase 1) to students in whole-class settings (n=128). Analyses included item difficulty and item discrimination. The rationale for retaining or rejecting tasks is presented along with psychometric/linguistic profiles documenting the evolution of example effective and ineffective prototype tasks. The final chapter of the report reflects on the lessons learned from the test development process we adopted and makes suggestions for further advances in this area.
Descriptors: Academic Discourse, Reading Comprehension, Textbooks, English (Second Language), Second Language Learning, Educational Testing, Program Effectiveness, Profiles, Psychometrics, Grade 4, Grade 5, Grade 6, Native Speakers, Discourse Analysis, Classroom Communication, Standards, Test Construction, Language Tests, Vocabulary, Grammar, Language Skills, Content Area Reading, Testing
National Center for Research on Evaluation, Standards, and Student Testing (CRESST). 300 Charles E Young Drive N, GSE&IS Building 3rd Floor, Mailbox 951522, Los Angeles, CA 90095-1522. Tel: 310-206-1532; Fax: 310-825-3883; Web site: http://www.cresst.org
Publication Type: Reports - Evaluative
Education Level: Grade 4; Grade 5; Grade 6
Sponsor: Institute of Education Sciences (ED)
Authoring Institution: University of California, Los Angeles, Center for the Study of Evaluation