This study examines the issues pertaining to South Korea's elementary physical education (PE) performance assessment, using an assessment literacy (Hay & Penney, 2013) perspective to propose future directions. Eight elementary teachers currently teaching PE were selected as participants. Data were collected through semi-structured in-depth interviews and on-site data analysis, and analyzed based on an inductive categorical analysis, the results of which were elicited from the four concept factors of assessment literacy. Four themes presented themselves during the research: first, teachers were unclear about the concept of assessment, making it difficult for them to carry out assessments effectively; second, assessments were conducted in a labor exchange and recycling manner, reducing their effectiveness; third, there was a lack of feedback; and fourth, there were teachers' critical thinking without pedagogy. These are the main problems in assessing PE in elementary schools. As for future directions, this study proposes the need for diversifying current teachers' education geared towards enhancing their assessment literacy capability, for providing on-site guidance to build students' assessment literacy, and for evaluating the assessment procedure.