Large-scale assessment programs, including alternate assessments based on alternate achievement standards (AA-AAS), must provide evidence of technical quality and validity. This study provides information about the technical quality of one AA-AAS by evaluating the standard setting for the science component. The assessment was designed to have three levels of cognitive difficulty, where each student is administered only the most appropriate level. The current study investigated panelists' levels of understanding during standard setting to explore whether there were differences based on cognitive level of the test. Data were collected at various time points through the use of open- and closed-ended questionnaires. Panelists reported that they understood the Performance Level Descriptor development and standard setting tasks and had common concerns and levels of understandings across levels of the AA-AAS.