The head of department position has been an integral role in the organisational structure of colleges and universities for over a hundred years. Recently, many institutions of higher education have called on department heads to provide advancing quality management and leadership to academic units in response to an increasingly complex and competitive post-secondary environment. To foster success in department heads' expanding roles, many universities are making adjustments to aspects of the position to ensure that it is structured in a way that will allow its incumbent to provide quality management and leadership. Yet, most institutions pay little attention to the evaluation of changes they have made to the head of department position. This paper examines an initiative at Mount Royal University in Canada, to develop an evaluation framework for assessing extensive changes that were made to the head of department position. The resulting metrics of the evaluation framework include academic leadership, job satisfaction, retention and leadership continuity, which are posited as "head of department success measures". Proposed methods of data collection to assess the metrics consist of on-line surveys, exit interviews of outgoing chairs and gathering of human resource statistics. The author uses the Mount Royal University experience as a case for illustrating possible implications for like-minded institutions. (Contains 1 figure and 1 table.)