We asked 22 right brain-damaged (RBD) patients and 11 elderly healthy controls to perform hand-pointing movements to free-field unseen sounds, while modulating two non-auditory variables: the initial position of the responding hand (left, centre or right) and the presence or absence of task-irrelevant ambient vision. RBD patients suffering from visual neglect, unlike RBD patients without neglect and healthy controls, showed a systematic rightward error in sound localisation, which was modulated by the non-auditory variables. Localisation errors were exacerbated by initial hand-position to the right of the body-midline, and reduced by the leftwards initial hand-position. Moreover, for the visual neglect patients, mere presence of ambient vision worsened localisation errors. These results demonstrate that although hand-pointing to sounds has often been considered a straightforward approach to investigate sound-localisation abilities in brain-damaged patients, in some patients it may actually reveal localisation deficits that reflect a combination of impaired spatial-hearing and spatial biases from other sensory modalities (i.e., vision and proprioception).