Although fathers are increasingly a focus of attention in research, there is a dearth of research on depressive symptoms among fathers, especially young fathers with toddlers. This study used longitudinal data to examine what risk factors, including the age status of fathers (e.g., late adolescence, emerging adulthood, and adulthood), may be associated with depressive symptoms of fathers when their children were 3 and 5 years of age. A subsample of families for which complete data were available on all variables was used in the analyses (n = 1,403). About 46% of study sample was African American, 27% White, 23% Hispanic, and 4% other race/ethnicity. Paternal depressive symptoms were measured using Composite International Diagnostic Interview-Short Form (CIDI-SF). Late adolescent fatherhood was significantly associated with third-year paternal depressive symptoms but not with fifth-year depressive symptoms. Those who reported low social support were more likely to be depressed at both times. Fathers who did not work for regular pay were more likely to be depressed at the third-year follow-up, but not at the fifth-year follow-up. Parenting stress and being booked/charged with a crime were not associated with third-year paternal depressive symptoms, but were with fifth-year paternal depressive symptoms. This study emphasizes the importance of screening for depressive symptoms of fathers even before the birth of their child and monitoring and treating postpartum depressive symptoms, as first-year depressive symptoms was a significant predictor for third- and fifth-year depressive symptoms. Service providers should focus on the mental health of fathers as well as mothers to promote healthy environments for their children.