The present research examined whether 12.5-month-old infants take into account what objects an agent knows to be present in a scene when interpreting the agent's actions. In two experiments, the infants watched a female human agent repeatedly reach for and grasp object-A as opposed to object-B on an apparatus floor. Object-B was either (1) visible to the agent through a transparent screen; (2) hidden from the agent (but not the infants) by an opaque screen; or (3) placed by the agent herself behind the opaque screen, so that even though she could no longer see object-B, she knew of its presence there. The infants interpreted the agent's repeated actions toward object-A as revealing a preference for object-A over object-B "only" when she could see object-B (1) or was aware of its presence in the scene (3). These results indicate that, when watching an agent act on objects in a scene, 12.5-month-old infants keep track of the agent's representation of the physical setting in which these actions occur. If the agent's representation is incomplete, because the agent is ignorant about some aspect of the setting, infants use the agent's representation, rather than their own more complete representation, to interpret the agent's actions.