A learning analytics dashboard enables teachers and students to monitor and reflect on their online teaching and learning patterns. This study was a review of prior studies on learning analytics dashboards to show the need to develop an instrument for measuring dashboard success. An early version of the instrument based on the framework of Kirkpatrick's four levels of evaluation was revised through expert reviews and exploratory factor analysis. The instrument contains five criteria: visual attraction, usability, level of understanding, perceived usefulness, and behavioral changes. The validity of the instrument was subsequently tested with factor analysis. A total of 271 samples from students who utilized a learning analytics dashboard for one semester were collected and analyzed using structural equation modeling. In the model with fair fit, the visual attraction and usability of the dashboard significantly affected the level of understanding, and level of understanding affected perceived usefulness, which in turn significantly affected potential behavior changes. The findings of this study have implications for designers who want to develop successful learning analytics dashboards, and further research is suggested related to measuring the cross validity of the evaluation instrument to broaden its usage.