Background: The use of metacognition is critical to learning, especially in fields such as engineering that involve problem-solving and difficult conceptual material. Due to limitations with current methodological approaches, new methods are needed to investigate engineering students' metacognitive engagement in learning situations that are self-directed, such as study groups. Purpose: Our purpose was to develop an approach to investigate the metacognitive engagement of undergraduate engineering students in self-directed learning environments. The Naturalistic Observations of Metacognition in Engineering (NOME) Observational Protocol and Coding Strategy is a qualitative data collection method that allows researchers to observe the behaviors of students who are studying in groups to determine the student's engagement in different metacognitive practices. The NOME is intended to be used by researchers interested in studying online metacognitive behaviors without the direct interference of a methodological approach. Design/Method: We observed three study groups where students were working on an engineering problem-solving homework assignment. Using a taxonomic definition of metacognition, we coded episodes of observation transcripts to identify behaviors that represented key definitions in the taxonomy. Results: We combined subcodes and descriptions of behaviors with key definitions to develop a coding strategy useful for future observational studies. Evidence of intercoder agreement and agreement in unitizing indicates that the coding strategy can reliably be used by multiple trained coders to identify metacognitive engagement. Conclusions: The reliability evidence shows that the NOME may be a useful tool for researchers in engineering education interested in studying the metacognitive habits of engineering students in self-directed study.