This study compared how three different gloss modes affected college students' L2 reading comprehension and vocabulary acquisition. The study also compared how results on comprehension and vocabulary acquisition may differ depending on the four assessment methods used. A between-subjects design was employed with three groups of Mandarin-speaking college freshmen. The participants read an English passage, each with different gloss formats: in-text mode, marginal gloss mode, and pop-up mode. The English proficiency level of the participants was elementary. Summary writing and multiple-choice questions were used to assess reading comprehension; word translation and word matching were used to assess vocabulary acquisition. Results of a multivariate analysis of covariance (with proficiency as the covariate) indicated that different assessment methods using the same gloss mode yielded different outcomes. In general, while the marginal gloss mode resulted in the highest scores on the multiple choice test, in-text glosses brought about the best performance in the rest of the tests, and pop-up glosses led to the lowest scores on all four tests. This study suggests that student performance outcomes concerning gloss mode may be assessment specific. Researchers should consider how investigators obtain their outcomes when interpreting the results of a study.