Comparing probabilities is a useful skill in life. Binary choice tasks are popular means in research on probabilistic reasoning. Falk, Yudilevich-Assouline, and Elstein ("Educational Studies in Mathematics," 81(2), 207-233 2012) noted that many of these tasks contain design flaws. We designed and evaluated an extended and improved binary choice item set. In each trial, children were shown two boxes containing desired and undesired elements and had to identify the best box to blindly draw from. We took into account four necessary item set characteristics: (un)desired elements in the correct box, total number of elements in the correct box, and difference between desired and undesired elements in the correct box. Furthermore, some extensions to Falk et al.'s design (2012) were implemented: items in which one box certainly provided a desired element, items with three competing colors of elements, and items with lower "countability" of the elements. Results showed that extensions added to the design did not imperil internal consistency and validity. Younger children were more likely to give correct answers when the correct box contained more desired elements and older children are better at comparing probabilities than younger ones overall.