Background: Social support has been identified as a major protective factor in preventing mental health problems and also as a major contributor to quality of life. People with intellectual disabilities (ID) have been identified as having limited social support structures. Interventions have been focused on promoting their social presence and integration. However, previous studies have shown that this does not always lead to the formation of social relationships. To date few studies have looked at how having an ID leads to impoverished social networks. This study aimed to do this by contrasting the social relationships of people with physical disabilities (PD) and people with ID. Methods: Two groups of participants were recruited; 30 people with mild ID and 17 people with PD. Social and functional support networks were assessed, in addition to life experiences. Between and within group differences were then explored statistically. Results: Adults with ID had more restricted social networks than PD, despite being involved in more activities. Social support for adults with ID was mainly provided by family and carers and few relationships with non-disabled people were identified. In contrast adults with PD had larger social networks than had been reported in the mainstream literature and had a balance of relationships with disabled and non-disabled people. Conclusions: The results suggest that there are additional processes attached to having an ID, which lead to continued impoverished lifestyles. The findings also endorse other work that suggests being physically integrated and engaged in a wide range of activities does not guarantee good social and emotional support.