This paper reports on the development of a system in which naturalistic inputs are collected by a web-based e-reader and, in combination with a measurement of readers' comprehension of that text, are analyzed by a neural network to determine the nature of the relationship between the annotations and comprehension. Results showed that neural networks can be trained to take naturalistic inputs, like textual annotations, and produce reasonably accurate predictions of a dependent variable. The potential application of this system as a method for formatively assessing the work of students in broader learning environments, such as corporate or governmental training environments, massive open online courses (MOOCs), and statewide standardized curricula are discussed.