This study tested whether multiple-object tracking-the ability to visually index objects on the basis of their spatiotemporal history-is scene based or image based. Initial experiments showed equivalent tracking accuracy for objects in 2-D and 3-D motion. Subsequent experiments manipulated the speeds of objects independent of the speed of the scene as a whole. Results showed that tracking accuracy was influenced by object speed but not by scene speed. This held true whether the scene underwent translation, zoom, rotation, or even combinations of all 3 motions. A final series of experiments interfered with observers' ability to see a coherent scene by moving objects at different speeds from one another and by distorting the perception of 3-D space. These reductions in scene coherence led to reduced tracking accuracy, confirming that tracking is accomplished using a scene-based, or allocentric, frame of reference.