This work investigates how people make judgments about the content of their visual working memory (VWM). Some studies on long-term memory suggest that people base those metacognitive judgments on the outcome of a retrieval attempt. In contrast, Son and Metcalfe (2005) observed that people identify poorly remembered items immediately, presumably by the lack of familiarity for the retrieval cue. We tested these two hypotheses in the context of metacognition in VWM. In three experiments, we investigated participants' response behavior in a color reproduction task with a hidden color wheel. With this procedure, participants must search for the intended response, starting from a random color. We assumed that instant awareness of the inability to retrieve an information would be reflected in selecting the first, random color, rather than search for a particular color in the wheel. Although participants provided a substantial number of low-confidence responses, results of an adapted mixture modeling analysis yielded little evidence for quick guesses. Rather, participants consistently searched for a color (even with unfamiliar retrieval cues in Experiment 2), and only quickly guessed when being cued with objects at test that were not previously presented (Experiment 3). We conclude that people usually engage in retrieval attempts for providing judgments about their VWM, even when information is poorly remembered.