Introduction: Education research has clearly verified that a student's perception of the system to evaluate the subject matter will play a fundamental role in his/her implication (deep approach vs. surface approach) in the teaching/learning process of the subject matter. The present work aims to examine the factorial validity and reliability of a scale to evaluate university students' perceptions when the academic year starts of the evaluation system of the subject matter to be taught. Method: A scale was created from debates organised with, and reflections made, in discussion groups during a seminar on education evaluation at the Universitat Jaume I. It was run by the Unitat de Suport Educatiu (Education Support Unit) in academic years 2012/13 and 2013/14, and 18 teachers participated who taught the various subject matters and degrees. A questionnaire was administered to a study sample of 435 university students who studied different matters and degrees. Results: The results from a confirmatory factorial analysis and Cronbach's alpha revealed that: a) the questionnaire had a three-dimensional structure; that is, it consisted in three independent factors that refer to three important constructs of the evaluative quality of learning; b) it had good psychometric properties for validity and reliability. Conclusion: The practical implications that can derive from using this instrument in the university context are discussed herein.