This article characterizes the advances, opportunities, and challenges for psychometrics of simulation-based assessments through a lens that views assessment as evidentiary reasoning. Simulation-based tasks offer the prospect for student experiences that differ from traditional assessment. Such tasks may be used to support evidentiary arguments that differ considerably from those typical in assessment. These novel assessment arguments are richer or more nuanced than those commonly used in terms of the targeted inferences about students, the evidence that facilitates those inferences, and the tasks that allow for the collection of such evidence. Driving principles of assessment are reviewed, and their implications for specifying student, task, and evidence models for simulation-based assessments are described. Potential pitfalls in these and related aspects of the assessment development process are described. Strategies for solutions to some of the more imminent challenges to psychometrics for simulations are discussed.