The public understanding of science is a term now considered somewhat narrow and carrying connotations of a "deficit model" of the public. Since the late 1980s, however, regular public surveys have been conducted to find out what facts about science the public knows. These include such questions as whether the Earth goes round the Sun, which is frequently quoted as an indication of public knowledge. The results of the surveys have been used as comparative measures of public understanding across time and across countries since their inception. Concurrently, however, there have been questions raised in the literature about the validity and meaning of these surveys. In this paper we review that literature. We also describe the results of administering the survey to over 500 scientists. The scientists found the questions problematic, especially if the topic of the question lay within their own discipline, and in many cases did not remember the correct answer. We discuss the implications of the findings for using such questions in public surveys and recommend that current assessments of public knowledge which use these questions be revised, recognising that such measures are not meaningful as indicators of public engagement with science.