Based on a socio-ecological framework for physical education (PE), this study explored and analysed the perspectives of physically active and inactive adolescent boys and girls with different socioeconomic status (SES), regarding the meanings that PE had on their physical activity (PA) lifestyles throughout childhood and adolescence. An interpretative and qualitative design was adopted and operationalised through an individual two-hours in-depth interview. Thematic analysis identified five main themes that distinguish active from inactive adolescents: 'Early experiences of PE at primary school'; 'PE experiences in middle and secondary school'; 'Friendly, professional and pedagogue. That's what I want from my PE teacher'; 'Friends in PE and PA'; 'Active lifestyles? Are schools and PE ready for making a difference …'. Regardless of the lifestyle, girls revealed more PE negative experiences, related to uninteresting and competitive activities, groups' constitution and logistic factors. SES played a major role on their PA and PE experiences, with low SES adolescents having limited opportunities inclusively within the primary school PE context. This study provides evidence for the importance of promoting positive PE and PA experiences in early years. Based on adolescents' voices, several recommendations are discussed that can be considered for improving PE professionals' effectiveness in promoting active lifestyles.