A common practice in educational assessment is to construct multiple forms of an assessment that consists of tasks with similar psychometric properties. This study utilizes a Bayesian multilevel item response model and descriptive graphical representations to evaluate the psychometric similarity of variations of the same task. These approaches for describing the psychometric similarity of task variants were applied to two different types of assessments (one traditional assessment and one performance-based assessment) with markedly different response formats. Due to the general nature of the multilevel item response model and graphical approaches that were utilized, the methods used for this work can readily be applied to many assessment contexts for the purposes of evaluating the psychometric similarity of tasks.