Recognition memory involves knowing an item was learned (familiarity) and remembering contextual details about the prior learning episode (recollection). We tested three competing hypotheses about the role of the hippocampus in recollection and familiarity. It mediates either recollection or familiarity, or serves both processes. We further tested whether the left temporal lobe mediates recollection and the right temporal lobe familiarity (modes of processing view), or whether the two temporal lobes mediate remembering material specifically (material specificity view). We investigated 24-h face recognition using the "remember-know" procedure. We studied 23 left and 24 right temporal lobe epilepsy (LTLE/RTLE) patients with and without hippocampal sclerosis (HS+/HS-) and 31 healthy participants. HS+ patients made fewer know responses than HS- patients or healthy participants. RTLE was related to fewer remember responses than LTLE. Our results suggest the hippocampus has a critical role in familiarity. Further, our findings support the material specificity hypothesis of laterality. (Contains 2 tables.)