Evaluation researchers and practitioners acknowledge that involving stakeholders in the planning and implementation of an evaluation increases buy-in, understanding, and use. With the recent increase in multi-site evaluations of large federal programs, evaluators must think differently about how to encourage meaningful collaboration by stakeholders. To date, there has been no published measure of such involvement, despite recent calls for more systematic, replicable research. The purpose of this study was to validate the Evaluation Involvement Scale for use in multi-site evaluations. Between the fall of 2006 and spring of 2007 data were collected from an electronic survey and phone interviews of evaluators and principal investigators of four National Science Foundation program evaluations. Using Messick's unitary concept of validity as a framework, theoretical, statistical, and rational evidence is provided to support the use of the Evaluation Involvement Scale to measure stakeholder involvement in multi-site evaluations. (Contains 5 tables and 1 figure.)