Compared with reading, writing and listening, there has been a paucity of empirical data documenting learners' experiences of speaking English as a second language (ESL) or English as a foreign language (EFL) in different learning contexts in spite of the fact that developing the ability to speak in a second or foreign language is widely considered a daunting task for most language learners. This paper reports on the findings of a questionnaire survey of the English speaking difficulties experienced by two Chinese populations: one group of university students from mainland China and one group of university students from Hong Kong. Drawing on various theoretical perspectives, the students' perceived difficulties are analysed in relation to linguistic deficiency, oral language processes, conversational skills and academic speaking conventions, affective influence as well as affordance of opportunities to use English for spoken communication in the students' learning contexts. Important implications of the results for pedagogical practices that foster development of ESL or EFL speaking skills are discussed. (Contains 7 tables and 1 note.)