This chapter addresses the psychometric challenges in assessing English language learners (ELLs) and students with disabilities (SWDs). The first section addresses some general considerations in the assessment of ELLs and SWDs, including the prevalence of ELLs and SWDs in the student population, federal and state legislation that requires the inclusion of all students in large-scale assessments, validity considerations in the assessment of ELLs and SWDs, importance of test accommodations in their assessment, and an introduction to the psychometric challenges, which are intricately interwoven with validity and fairness considerations, in assessing ELLs and SWDs. The second section discusses the efficacy of test accommodations and modifications for SWDs and ELLs. The third section addresses the need for invariant measurement for ELLs and SWDs. The last section addresses issues related to including SWDs and ELLs in measures of "growth" and of educator effectiveness.