The purposes of this study were to (a) explore whether early fractions understanding at 4th grade is differentially challenging for students with versus without adequate whole-number competence and (b) identify specific whole-number skill associated with difficulty in fractions understanding. Based on initial whole-number competence, 1,108 4th graders were classified as having (a) adequate whole-number competence (n = 775), (b) less severe whole-number difficulty (n = 201), and (c) severe whole-number difficulty (n = 132). At the end of 4th grade, they were assessed on fractions understanding and further classified as with versus without difficulty in fractions understanding. Multi-level logistic regression indicated that compared to students with adequate whole-number competence, those with less severe whole-number difficulty were almost 5 times as likely to experience difficulty with fractions understanding whereas those with severe whole-number difficulty were about 32 times as likely to experience difficulty with fractions understanding. Students with severe whole-number difficulty were about 7 times as likely to experience difficulty with fractions understanding compared to those with less severe whole-number difficulty. Among students with adequate whole-number competence, the pretest whole-number skill distinguishing those with versus without difficulty in fractions understanding was basic division facts (i.e., 2-digit dividend ÷ 1-digit divisor) and simple multiplication (i.e., 3-digit × 1-digit without regrouping). The role of whole-number competence in developing initial fractions understanding and implications for instruction are discussed.