Critical thinking is a learning outcome highly valued by higher education institutions and the workforce. The Educational Testing Service (ETS) has designed a next generation assessment, the HEIghten™ critical thinking assessment, to measure students' critical thinking skills in analytical and synthetic dimensions. This paper introduces the theoretical framework that guided the assessment design, and also reports on the preliminary validity evidence of the pilot data from over 3000 students from 35 two and four-year institutions. The critical thinking scores demonstrated satisfactory total and subscale reliabilities, were reasonably correlated with SAT scores, high school grade point average (GPA), and college GPA, and were able to detect cross-sectional performance difference between freshmen and seniors. In addition, most examinees reported having tried their best when taking the test. Results show that test-taking motivation has a significant impact on performance. We encourage institutions to pay attention to motivational issues in implementing low-stakes learning outcomes assessment such as the HEIghten™ critical thinking assessment.