NotesFAQContact Us
Search Tips
ERIC Number: ED539081
Record Type: Non-Journal
Publication Date: 2009-Jul
Pages: 10
Abstractor: As Provided
Reference Count: 8
Determining the Significance of Item Order in Randomized Problem Sets
Pardos, Zachary A.; Heffernan, Neil T.
International Working Group on Educational Data Mining, Paper presented at the International Conference on Educational Data Mining (EDM) (2nd, Cordoba, Spain, Jul 1-3, 2009)
Researchers who make tutoring systems would like to know which sequences of educational content lead to the most effective learning by their students. The majority of data collected in many ITS systems consist of answers to a group of questions of a given skill often presented in a random sequence. Following work that identifies which items produce the most learning we propose a Bayesian method using similar permutation analysis techniques to determine if item learning is context sensitive and if so which orderings of questions produce the most learning. We confine our analysis to random sequences with three questions. The method identifies question ordering rules such as, question A should go before B, which are statistically reliably beneficial to learning. Real tutor data from five random sequence problem sets were analyzed. Statistically reliable orderings of questions were found in two of the five real data problem sets. A simulation consisting of 140 experiments was run to validate the method's accuracy and test its reliability. The method succeeded in finding 43% of the underlying item order effects with a 6% false positive rate using a p value threshold of less than = 0.05. Using this method, ITS researchers can gain valuable knowledge about their problem sets and feasibly let the ITS automatically identify item order effects and optimize student learning by restricting assigned sequences to those prescribed as most beneficial to learning. (Contains 4 figures, 5 tables, and 3 footnotes.) [Additional funding for this paper was provided by the U.S. Department of Education's Graduate Assistance in Areas of National Need (GAANN). For the complete proceedings, "Proceedings of the International Conference on Educational Data Mining (EDM) (2nd, Cordoba, Spain, July 1-3, 2009)," see ED539041.]
International Working Group on Educational Data Mining. Available from: International Educational Data Mining Society. e-mail:; Web site:
Publication Type: Reports - Research; Speeches/Meeting Papers
Education Level: Grade 10; Grade 11; Grade 12; Grade 7; Grade 8; Grade 9; High Schools; Junior High Schools; Middle Schools; Secondary Education
Audience: N/A
Language: English
Sponsor: Institute of Education Sciences (ED); Office of Naval Research (ONR); Spencer Foundation; National Science Foundation
Authoring Institution: International Working Group on Educational Data Mining