We surveyed researchers in the health and exercise sciences to explore different areas and magnitudes of bias in researchers' decision making. Participants were presented with scenarios (testing a central hypothesis with p = 0.06 or p = 0.04) in a random order and surveyed about what they would do in each scenario. Participants showed significant bias in deciding to randomize additional participants when p = 0.06 (24.5%) compared to when p = 0.04 (6.5%), ?[superscript 2] = 19.86, odds ratio (OR) = 13.5. Based on this bias to randomize additional participants, we present statistical simulations showing the cumulative negative effect this bias has on the false positive rate for a field. While it is important for researchers to have flexibility in their data-processing decisions, there is a cost/benefit trade-off to experimenter degrees of freedom and it is important that these decisions be unbiased. These data highlight the importance of a priori analysis plans in research.