School self-evaluation (SSE), as an important leverage for quality assurance, often relies on surveys among staff members to collect information on the schools' functioning. The extent to which respondents cognitively process items as developers intended them determines the cognitive validity of SSE results. However, it is unclear what problems occur in respondents' cognitive processes which lead to cognitively invalid SSE results and how respondents' positions in the school affects these cognitive processes. This study draws on cognitive interviews conducted with 20 teachers and principals to understand their thinking process while answering an SSE survey. Cognitively invalid results were analysed using a content analysis to identify problems in respondents' cognitive processes. Findings showed that respondents experience semantic and syntactical issues when interpreting items. While elaborating, problems were found regarding items' topic and focus, particularly concerning whom to make a statement about. Issues also emerged in the response stage, especially that the "don't know" option was not used as intended. Respondents' positions influence their understanding about whom a statement is required and how self-evident some items are to them. These problems should be taken into account by developers of SSE surveys and other instruments that intend to measure organisational characteristics.