Wilderness therapy (WT) provides an alternative treatment modality for a number of mental health issues. It holds particular appeal for at-risk youth, a population that is often less responsive to traditional psychotherapeutic interventions. Anecdotal accounts on the effectiveness of WT often show positive outcomes. Still, some researchers have questioned the value of WT outcome studies on the basis of dubious methodology. The current study sought to address this issue through its use of well-validated observational scales drawn from the Child Behaviour Checklist (CBCL). While on a canoe expedition, three trained raters assessed the conduct of a group of at-risk youth aged 15 to 18 years. Results showed very little inter-observer agreement (Cronbach's alpha = 0.12). The findings underline the challenges of measuring WT outcomes. Even when equipped with validated and behaviourally anchored instruments, there appears to be little consistency among observers in terms of the degree to which WT influences participant change. The study demonstrates that when structured observation of WT participants is undertaken, outcomes are less definitive. This supports the criticisms offered by previous researchers who have expressed that in the absence of improvements to WT research methods, outcome findings are likely to be of questionable validity. (Contains 1 table.)