The effect of different combinations of item response theory (IRT) item parameters (item difficulty, item discrimination, and the guessing probability) on the reliability and construct validity (correlation with the latent trait being measured) of pretest, posttest, and gain scores was analytically examined using the 3-parameter logistic (3PL) model (Birnbaum, 1968). The analyses were performed at the single-item level using items with specified combinations of item parameters. In general, the pretest item validities and reliabilities were much higher than those of the gain score. It is also the case that the reliability of the gain score is at a minimum when that of the pretest score is at a maximum, namely, near the middle of the distribution of item difficulty (i.e., for moderately difficult tests). Furthermore, using IRT we show that the regression of the gain score on the latent trait being measured is nonmonotonic, meaning a person at a given low ability level has the same expected gain score as one of the corresponding high ability. These results not only are of theoretical interest but also may be of interest to applied practitioners and test developers.