We present an evaluation of the impact of an atoms-first curriculum on student success in introductory chemistry classes and find that initially a lower fraction of students obtain passing grades in the first and second quarters of the general chemistry series. This effect is more than reversed for first-quarter students after one year of instruction experience with the atoms-first curriculum, yet challenges with the second quarter remain. This study involves more than 8000 students over five years, with two-fifths of them taught using the atoms-first approach. It provides a cautionary example for the adoption of this approach.