The trouble with mathematics is that it looks a little more logical and consistent than it is. Mathematics has a universally recognised exactitude. It also has an inexactitude that tends to remain concealed. This author contends that the trouble with mathematics is that it is neither structured, nor unstructured. It is nearly structured, but not fully. It is neither tamed nor wild. It is both nearly tamed and forever untameable. The author uses the jazz metaphor as a useful way of understanding complexity, and explains that his analysis of the jazz metaphor and its relation to mathematics teaching has led to the identification of five key components of jazz: (1) structure; (2) improvision; (3) playing outside; (4) pursuit of ideals; and (5) "ways of the hand". This article concentrates on the third component: "playing outside", which, in jazz refers to a radical form of improvisation that deliberately transcends the established structures. Playing outside is essential in the study of mathematics, where proof is not enough for establishing mathematical truth.