This article documents collective memories of the founding, curriculum, and attendees of one of the first (1866) Reconstruction Era Quaker-Freedmen School sites in the Southeastern United States. It applies critical oral history methodology including the collection of primary documents, previous investigations into the school, and interviews of community elders. Through the close study of the school's history, including the present quest for official historical memorialization, this investigation accentuates how Whiteness as property remains across generations and contexts. What began as a historical investigation of the school necessarily evolved as an analysis of the complications in race relations in the observed college town.