This paper discusses academics' perceptions of challenges during the implementation of peer observation of teaching (POT) for promoting teachers' reflection and teaching improvement. This qualitative case study involved 11 academics teaching English at a university in Vietnam. Data were analysed inductively and thematically. Several challenges were identified: limited learning from observing junior peer teaching and junior peers' feedback, uncritical feedback from peers without closeness, lack of sensitivity in giving feedback, and limited time for POT. These findings suggest that a strong peer relationship in a Confucian culture appears to be key for successful POT.