Two broad theories of visual working memory (VWM) storage have emerged from current research, a discrete slot-based theory and a continuous resource theory. However, neither the discrete slot-based theory or continuous resource theory clearly stipulates how the mental commodity for VWM (discrete slot or continuous resource) is allocated. Allocation may be based on the number of items via stimulus-driven factors, or it may be based on task demands via voluntary control. Previous studies have obtained conflicting results regarding the automaticity versus controllability of such allocation. In the current study, we propose a two-phase allocation model, in which the mental commodity could be allocated only by stimulus-driven factors in the early consolidation phase. However, when there is sufficient time to complete the early phase, allocation can enter the late consolidation phase, where it can be flexibly and voluntarily controlled according to task demands. In an orientation recall task, we instructed participants to store either fewer items at high-precision or more items at low-precision. In 3 experiments, we systematically manipulated memory set size and exposure duration. We did not find an effect of task demands when the set size was high and exposure duration was short. However, when we either decreased the set size or increased the exposure duration, we found a trade-off between the number and precision of VWM representations. These results can be explained by a two-phase model, which can also account for previous conflicting findings in the literature.