Based on three surveys carried out for studying living conditions of youth, women and elderly living in six remote areas (Tuen Mun, Yuen Long, Tin Shui Wai, Sheung Shui, Fan Ling and Tai Po) in the New Territories of Hong Kong, this paper reports the poverty and social exclusion of these three groups of people. The quality of life of youth, women and elderly is adversely affected by limited job opportunities, high cost of travel for employment, and poor neighbourhoods in the community. However, perceptions of reasons for their problems are different for the three groups, due to differences in bonding and bridging social capital they have, as well as the differences in perceptions on social exclusion and discrimination directed upon them.