In this study, we present a new instrument, the Participation and Engagement Scale (PES), for the evaluation of the students' involvement in STEM-oriented activities. The instrument was administered to about 1000 secondary school students who participated in the activities of the Italian "Piano Nazionale Lauree Scientifiche" in Biology, Chemistry and Physics. The activities were carried out in a remote modality due to the COVID-19 restrictions. Through an exploratory and confirmatory factor analysis, it was possible to validate a two-factor structure of the instrument: satisfaction towards the activities and value of the activities. The proposed factor structure shows a good model fit, with each of the obtained scales displaying excellent reliability. Construct-related validation evidence was obtained through the Rasch analysis, which allowed further psychometric improvement of the instrument. Convergent validation evidence was established through a correlation with the academic motivation and perceived difficulties scales. Using the proposed instrument, we found no statistically significant relationships between engagement, the different types of science activities attended, and the intention to enroll in a STEM course. Implications of the study for the evaluation of public understanding of science activities in both remote and in-presence modalities are also discussed.