Purpose: This study examined use of a speaker's direction of gaze during word learning by boys with fragile X syndrome (FXS), boys with nonsyndromic autism spectrum disorder (ASD), and typically developing (TD) boys. Method: A fast-mapping task with follow-in and discrepant labeling conditions was administered. We expected that the use of speaker gaze would lead to participants selecting as the referent of the novel label the object to which they attended in follow-in trials and the object to which the examiner attended in the discrepant labeling trials. Participants were school-aged boys with FXS (n = 18) or ASD (n = 18) matched on age, intelligence quotient, and nonverbal cognition and younger TD boys (n = 18) matched on nonverbal cognition. Results: All groups performed above chance in both conditions, although the TD boys performed closest to the expected pattern. Boys with FXS performed better during follow-in than in discrepant label trials, whereas TD boys and boys with ASD did equally well in both trial types. The type of trial administered first influenced subsequent responding. Error patterns also distinguished the groups. Conclusion: The ability to utilize a speaker's gaze during word learning is not as well developed in boys with FXS or nonsyndromic ASD as in TD boys of the same developmental level.