Both in physics education and in science education more generally concerns exist that formal K-12 education structures limit and, in some cases, diminish students' interest and agency in these fields. Many stakeholders have turned to informal learning experiences as a means to inspire young people to pursue continual learning in these fields in ways that foster creativity and self-determination. While research exists on the effect of these informal science experiences on students' science identities and broader science, technology, engineering, and math (STEM) identities, little is known about how specific informal science education experiences relate to students' "physics identity"--a construct strongly associated with physics career choice. The current study contributes to the literature by examining the effect of several informal science experiences on students' physics identity. Drawing on data from a national survey administered to students in required English courses at 27 colleges and universities across the US (N=15 847), we used multiple regression to test the relationship between informal science experiences in various topical areas at two educational levels (K-8 and 9-12) and students' physics identity, while controlling for science background and demographics. The results reveal positive effects for stereotypic informal experiences in physical science (e.g., tinkering, competitions) as well as for talking science with friends or family. In addition, there were negative relationships between biology-related experiences (at both levels) and physics identity. Group comparisons further revealed that female students were more likely to report participating in biology-related activities and less likely to report participating in tinkering, STEM competitions, and talking science with friends or family. Students who identified themselves as Black or Hispanic were also less likely than those of other racial or ethnic groups to report tinkering and talking science with friends or family. We use this evidence to build the case that informal learning experiences in physics should move beyond stereotypic activities, increase accessibility, facilitate discourse with family or friends, and focus on interdisciplinary experiences that better engage young participants with a wide range of interests that are connected to physics.