This study aims at developing a comprehensive teaching evaluation system, a more useful educational technology, for achieving relatively reliable and valid results that can be well acknowledged by instructors, students, and by administrators. Adopting multi-method approaches, the study integrates student evaluation, expert evaluation and regular examination of teaching into teaching rating. Especially, a novel method of "evaluation on the same platform", as an important part of the comprehensive evaluation system, is designed as a fair approach for teaching evaluation. All the parameters in the proposed system are determined by implementing the Analytic Hierarchy Process to get the weight of each factor. Empirical studies are implemented in the experimental university for more than six semesters. The result demonstrates that the evaluation system may be more reliable if performance of the instructors is assessed by more comprehensive raters. The whole evaluation system is implemented conforming to a series of serious procedures to ensure democracy, fairness and effectiveness.