We studied the impact of metacognitive reflections on recently-completed work as a way to improve the retention of newly learned problem-solving techniques. Students video recorded themselves talking through problems immediately after finishing them, completed ongoing problem-solving strategy maps or problem-sorting exercises, and filled out detailed exam wrappers. We assessed students' problem-solving skillfulness using a combination of validated instruments and final exam questions scored using a rubric that targets problem-solving best practices. We found a small but significant difference between the rubric score distributions for the control and treatment groups. However, a multiple ordered logistic regression using treatment and Force Concept Inventory (FCI) pretest score as predictors showed that this difference is better explained by the latter. The surprising impact of conceptual preparation on problem-solving skill suggests two things: the importance of remediation for students with insufficient conceptual understanding and the need to consider problem-solving interventions in the context of students' conceptual knowledge base.