Dominating strands within the research literature on second homes explain social conflicts between rural hosting and visiting second home populations by describing their differing perspectives on rural development. Such presentations suggest that locals are likely to welcome new developments in order to enhance the economic viability of their rural communities, whereas second home owners advocate conservation of the rural idyll that attracted them in the first place. In this paper, we argue that these simplified differences conceptualised in the contemporary second home research literature are faulty. By analysing 42 qualitative in-depth interviews with second home owners and locals in four Norwegian municipalities, we demonstrate how both locals and second home owners are protective of their rural idyll and, at the very same time, open to rural development. More exactly, locals and second home owners alike generally welcome new activities only when they do not take place in their own vicinity. As such, both categories' interests reflect a "Not in my backyard" (NIMBY) line of logic. Hence, we argue that the major lines of conflict concerning land use in second home municipalities do not run between locals and visitors but between those initiating different kinds of new developments and those appreciating the hitherto existing qualities and appearance of the areas of development localisation. Nevertheless, the influx of second home owners is still influencing the potential for land use conflicts due to the high number of actors present in the same location. In effect, crowding a rural area with second home developments generates more "backyards" and thus guardians of these. (Contains 1 table and 1 figure.)