Borrowing from a clinical psychology observational methodology, thin-slice observations were used to assess autism characteristics in toddlers. Thin-slices are short observations taken from a longer behavior stream which are assigned ratings by multiple raters using a 5-point scale. The raters' observations are averaged together to assign a "thin-slice" value for each observation. In this study, a total of 60 toddlers were selected from a video archive: 20 children with typical development, 20 children with developmental language disorder, and 20 children with autism. In the first part of this study, 20 raters observed small play segments between toddlers and an assessor. Raters assigned scores to each of the 60 toddlers on items related to autism symptomatology. Item analysis and generalizability and decision studies were conducted to determine the factor structure and optimal number of raters to achieve a stable estimate of autism characteristics. In the second part of the study, generalizability and decision studies were conducted to determine the most efficient and optimal combination of raters and naturalistic contexts. This pilot study provides recommendations for optimizing the utility of thin-slice observations for measuring autism symptomatology in young children.