Accurately judging one's performance in the classroom can be challenging considering most students tend to be overconfident and overestimate their actual performance. The current work draws upon the metacognition and decision making literatures to examine improving metacognition in the classroom. Using historical data from several semesters of an upper-level undergraduate course (N = 127), we analyzed students' judgments of their performance and their actual performance for two exams. Students were instructed on the concepts of overconfidence, received feedback on exams, and were given incentives for accurate calibration. We found results consistent with the "unskilled and unaware" effect Kruger & Dunning ("Journal of Personality and Social Psychology," 77(6), 1121-1134, 1999) where lower performing students initially displayed overconfidence and the highest performing students initially displayed underconfidence. Importantly, students were able to change both judgments and performance such that metacognitive accuracy improved significantly from the first to the second exam. In a second study, two additional semesters for the same course used in Study 1 were examined (N = 90). For one of the semesters feedback was not provided, allowing us to determine whether feedback can improve both metacognitive judgments and performance. Our findings revealed significant improvements in performance paired with decreases in overconfidence on Exam 2, but only for students who received feedback about their performance and judgments. We postulate that feedback may be an important component in improvement metacognitive judgments.