The data collection of a cohort study requires making many decisions. Each decision may introduce error in the statistical analyses conducted later on. In the present study, a procedure was developed for estimation of the error made due to the composition of the sample, the item selection procedure, and the test equating process. The math results for 15-year-olds in the Dutch cohort study COOL[superscript 5-18] were used to illustrate how to account for the different sources of error in a cross-cohort analysis. Results showed the error variances to be severely underestimated if they were calculated under the common assumption of a simple random sample. The impact of ignoring error variance clearly manifested when the students' mathematical achievements were examined. Whereas analyses under the assumption of a simple random sample resulted in a significant downward trend in achievement, alternative analyses showed this conclusion not to be solidly supported by the data.