Reliability of direct observation outcomes ensures the results are consistent, dependable, and trustworthy. Typically, reliability of direct observation measurement approaches is assessed using interobserver agreement (IOA) and the calculation of observer agreement (e.g., percentage of agreement). However, IOA does not address intraobserver agreement, or the withinrater agreement, analogues to internal consistency. One approach to assess both IOA and intraobserver agreement is generalizability theory. This paper describes generalizability theory and provides a case study example of using generalizability theory with direct observation measures in emotional and/or behavioral disorders research.