Scholastic tests regard cognitive abilities to be domain-specific competences. However, high correlations between competences indicate either high task similarity or a dependence on common factors. The present rating study examined the validity of 12 Programme for International Student Assessment (PISA) and Third or Trends in International Mathematics and Science Study (TIMSS) tasks. Two tasks per competence (reading, mathematics, science, problem solving) from PISA and TIMSS were assessed by 34 teachers and 33 psychology students on 11 scales: difficulty, curriculum reference, knowledge versus thinking, reading competence, verbal ability, math competence, science competence, problem solving, reasoning, general knowledge, and intelligence. Intraclass correlation between two randomly chosen raters was r[subscript ic] =0.59. None of the tasks represented the intended target competence concisely. In five PISA tasks, competences other than those intended were seen as being more relevant. TIMSS tasks were seen as more curriculum-related and requiring more school knowledge than PISA tasks. For solving PISA tasks, thinking/reasoning ability and general intelligence were rated as being more important (d = 0.36). Only small differences were found between students' and teachers' ratings.