This paper presents the early investigations about the nature of sound of the Pythagoreans, and how they started a tradition that remains valid up to present times--the use of numbers in representing natural reality. It will touch on the Pythagorean notion of musical harmony, which was extended to the notion of universal harmony. How the Pythagorean ideas have inspired many great works in physics, such as those of Galileo, Kepler and Newton, will also be presented. In exploring the legacy of Pythagoras to physics and the study of the universe, some valuable insights on the nature of science that can inspire budding physicists are extracted.