In this theoretical essay, I argue that the current incidences of backlash to diversity are best understood as a dynamic of complicated, historic and intertwined desires for racial diversity and white entitlement to property. I frame this argument in the theories of critical race theory and settler colonialism, each of which provide necessary but incomplete analytic tools for understanding systemic racism and property rights. Situating universities and colleges as white settler property established on seizure contextualizes both the ways in which the desire for diversity is connected to white supremacy and leads to subsequent backlash to the presence of people of color, particularly those in positions of authority. I close with a discussion of the tension between property rights and potential cultural transformation.