Open courses have received a lot of attention in the last two years; however, the question of whether they serve learners has yet to be determined. This paper explores the challenges and potential in assessing the impact of open educational initiatives, particularly those that produce and share Open Educational Resources (OER). We use a collaborative international project as a case study to explore this issue. Bridge to Success was supported as part of the Next Generation Learning Challenges (NGLC) programme to work with a range of community colleges and other organisations in the US through monitored pilots. The project adapted existing course materials in mathematics and learning/personal development skills and released these as OER. A range of approaches were then used to assess the impact of the materials across a diverse set of users, combining data gathered from interviews and questionnaires with both educators and learners and from instructor rating of performance and related student results. This approach allows different indicators of performance to be brought together and so demonstrate the value of OER. However, our findings also highlight tensions between applying robust research methodologies in situations of open use with diverse stakeholders. We provide reflections and suggestions for ways forward in addressing the particular characteristics of openness and how they affect research, and how the multiple perspectives on what constitutes impact can be addressed.