The present study examined, within a longitudinal family-informed design and across middle childhood, the predictive associations between preference for solitude and social wariness, two forms of social withdrawal, and peer difficulties. Specifically, preference for solitude, rather than social wariness, was expected to predict peer victimization and rejection, two aspects of peer difficulties. A total of 1,014 children from the Quebec Newborn Twin Study were assessed by teachers and peers at ages 6, 7, and 10 years. Multilevel analyses conducted across three levels, between family, within family, and within person, revealed that preference for solitude, rather than social wariness, increased the risk for peer difficulties in terms of both peer victimization and peer rejection. Specifically, preference for solitude was systematically associated with peer rejection starting at age 6 years and became progressively associated with peer victimization over time. This pattern was found both between and within families. In addition, the predictive association with peer rejection was found within genetically identical, monozygotic twin pairs, suggesting that this predictive association existed after taking into account genetic vulnerabilities. Social wariness was systematically unrelated to peer difficulties. These findings suggest that preference for solitude, rather than social wariness, is a risk factor for peer difficulties. They underscore the relevance of distinguishing these dimensions of social withdrawal and illustrate the usefulness of a family-informed design to document the processes underlying childhood social adjustment.