Letters of recommendation are commonly used to assess the potential of undergraduate students to be successful undergraduate research assistants/interns or their potential as graduate students. However, there is evidence to suggest that reference letters can include unconscious (or implicit) bias that can affect decisions and limit opportunities for under-represented minorities and students from non-research institutions. This study uses a text analysis software program to examine 457 letters of recommendation for undergraduate students applying to undertake international research experience to determine whether there is a statistically significant difference in the language used to describe the students accepted into the programme (n = 36 letters) compared to those who were not accepted (n = 421 letters). Results suggest that letters of recommendation for the accepted students describe the productivity of the students with greater certainty and include a greater number of quotes from student work. In comparison, the letters for those students who were not accepted into the programme include more positive emotion and describe the insight of the student, but include more words associated with discrepancy and tentative statements. Despite no statistically significant differences in grade point averages, a similar pattern was observed between male and female applicants, white and non-white applicants, and applicants from research and non-research institutions. Results suggest a need to standardise letters of recommendation to ensure that the biases are minimised and do not present a barrier to increasing diversity in undergraduate research.