With technology at the fingertips of most undergraduate students, it has been difficult for instructors to fully engage students in the classroom, which has resulted in the creation of several innovative online cognitive assessment tools. These tools often integrate several cognitive learning strategies within an assessment, with the goal of actually enhancing learning, as opposed to just measuring it. In the current study, students' level of engagement and test performance using a recently developed online application were compared to their final multiple-choice paper-and-pencil exam mark to determine the efficacy of the new online application in achieving improved learning outcomes. Results indicated that students had high test scores using the online tool despite their limited engagement in the cognitive learning features, calling into question the online cognitive assessment tool's facilitation of long-term learning. Implications and recommendations for future online cognitive assessment application implementation in educational environments are discussed.