Purpose: The purpose of this article is to examine the utility of framing formative feedback to improve school leadership with a focus on task-based evaluation of distributed leadership rather than on role-based evaluation of an individual leader. Research Methods/Approach: Using data from research on the development of the Comprehensive Assessment of Leadership for Learning, we examine ways to design formative evaluation and feedback organized around distributed leadership practices. This study draws on qualitative data from iterative design research conducted with middle and high school principals, assistant principals, teachers, and staff in 2011. Findings: Many challenges in providing actionable, multirater feedback were addressed by using an assessment instrument that focused on measuring distributed instructional leadership practices. Users reported that task-based multirater feedback provided transparency in communicating a clear theory of action for school improvement and fostered formal and informal conversations around school improvement. Implications for Research and Practice: The study suggests that focusing on distributed leadership practices may help overcome some of the limitations to the use of evaluation feedback that is targeted to an individual leader. Key features of formative feedback design desired by school principals included transparency in the theory of action underlying assessment items to prime teachers and leaders for collaborative discussions of current and potential practices, and research-based guidance on next steps schools can take to build distributed instructional leadership capacity in their schools.