Many human languages have complex grammatical machinery devoted to temporality, but very little is known about how this came about. This paper investigates how people convey temporal information when they cannot use any conventional languages they know. In a laboratory experiment, adult participants were asked to convey information about simple events taking place at a given time, in spoken language and in silent gesture (i.e., using only gesture and no speech). It was shown that in spoken language, participants formed utterances according to the rules of their native language (Dutch), but in silent gesture, the temporal information was presented initially, and structurally separately, from the other information in the utterance. The experimental results are consistent with findings from natural systems emerging in situations of communicative stress: unsupervised adult second language learning and homesign. This confirms that presenting temporal information separately and initially (directly mirroring how temporal and propositional information can be represented semantically) is a robust strategy to talk about past and future when only sparse communicative means are available.