In an article published in "Educational Measurement: Issues and Practice," Gaertner and McClarty (2015) discuss a college readiness index based, in part, on nonacademic or noncognitive factors measured in middle school. Such an index is laudable as it incorporates important constructs beyond academic achievement measures that may be important in assessing college readiness. It also holds potential in identifying areas of need at a critical time and can theoretically inform important interventions that may be necessary prior to the later stages of high school. However, one of the noncognitive factors--motivation--appeared to be theoretically incomplete, underrepresented, and contained items irrelevant to the construct of academic motivation. We offer our concerns about the validity of this factor as measured by the index and the associated implications about the consequences of using this to inform educational interventions. Through this commentary, we hope to offer constructive feedback on future research that can build upon Gaertner and important work.