We briefly discuss the philosophical basis of science, causality, and scientific evidence, by introducing the hidden but most fundamental principle of science: the similarity principle. The principle's use in scientific discovery is illustrated with Simpson's paradox and other examples. In discussing the value of null hypothesis statistical testing, the controversies in multiple regression, and multiplicity issues in statistics, we describe how these difficult issues should be handled based on our interpretation of the similarity principle.