Peer review of research programmes is changing. The problem is discussed through detailed study of a selection process to a call for collaborations in the energy sector for the European Institute of Innovation and Technology. The authors were involved in the application for a Knowledge Innovation Community. Through the analysis of the case the article discusses the role of researchers acting in their new role of reviewers in situations where a number of important decision-making dimensions are cut out from the formerly direct influence on the researcher's quality assessment. In connection with this discussion the article will provide input to a critical review of the use of quantified scaling developed to systematize a quality assessment on the top of the peer review-based assessments. In addition the article discusses the challenges to the peer review system in Mode 2 type science and in cross-disciplinary research collaborations. This is a contribution to the assessment of new roles of peer reviewers in large policy decision-making systems like research funding, and an input to further discussions.