The experience of childhood family adversity may affect adolescent sexual behavior, yet the variation in the types of adversities is unknown. We analyzed data from the Taiwan Education Panel Survey. In 2007, 18,995 students aged 18 years retrospectively reported the age of first sexual activity and the experiences of childhood adversities. Multinomial logistic regression was used to examine the association of childhood adversities with the timing of first sexual occurrence. Parental psychopathology and parental death had significant associations with first sex to occur during periods before age 12, ages 12 to 15, and ages 15 to 18. The impact of some types of adversities on sexual behavior started as early as elementary school, but others influenced children's sexual behavior at a later stage. Parental separation/divorce showed a stronger association with early onset of sex activity for females, that is, sexual debut before age 12.