Faculty mentorship is thought to be a linchpin of graduate education in STEM disciplines. This mixed-method study investigates agreement between student mentees' and their faculty mentors' perceptions of the students' developing research knowledge and skills in STEM. We also compare both assessments against independent ratings of the students' written research proposals. In most cases, students and their mentors identified divergent strengths and weaknesses. However, when mentor-mentee pairs did identify the same characteristics, mentors and mentees disagreed about the mentee's abilities in 44% of cases in the Fall semester and 75% of cases in the Spring semester. When compared against performance-based assessments of mentees' work, neither faculty mentors' nor their mentees' perceptions aligned with rubric scores at rates greater than chance in most categories.