Increased access to the Internet has dramatically increased the sources from which students can deliberately or accidentally copy information. This article discusses our motivation to design, implement, and deploy an Internet based plagiarism detection system, called PAIRwise, to address this growing problem. We give details as to how we detect plagiarism and the various development phases we went through before releasing PAIRwise as an open source software tool. Our major conclusion is that it is possible to deploy an automated plagiarism system that is an effective, customizable, and affordable tool for investigating how plagiarism affects academic integrity. (Contains 7 figures.)