Numerous studies have been undertaken to design, develop, and provide validity evidence for using instruments to measure students' attitudes toward STEM (Science, Technology, Engineering, and Mathematics). This study presents validity evidence of scores produced from the S-STEM measurement tool and used to evaluate changes in attitudes during an educational intervention in a middle school robotics learning environment. All data were collected from middle school students who were involved in a school district-wide effort for integrating educational robotics into the classroom. Findings from this study provided not only internal structure validity evidence, but also criterion-related validity evidence of the proposed S-STEM tool use. In addition, measurement invariance results revealed that items in the S-STEM had equivalence in statistical properties of measurement across groups (e.g., grade level). The study provides further evidence that S-STEM survey is a powerful and useful tool to evaluate student attitude changes during STEM educational programs; offers suggestions for its future implementation; and presents other inspiring ideas for future STEM instrument development.