A computational cognitive modeling approach to understand test-takers’ strategy use in drag-and-drop math questions
- Burcu Arslan, Cognitive and Technology Sciences Center, Educational Testing Service, Princeton, New Jersey, United States
- Yang Jiang, Educational Testing Service, Princeton, New Jersey, United States
- Tao Gong, Educational Testing Service, Princeton, New Jersey, United States
- Madeleine Keehner, ETS, Princeton, New Jersey, United States
- Irvin Katz, Cognitive, Accessibility, and Technology Sciences Center, Educational Testing Service, Princeton, New Jersey, United States
AbstractComputer-based educational assessments often include questions with a drag-and-drop response. Logged data obtained from drag-and-drop responses allow us to go beyond scores, investigating the response strategies test-takers use to reach an answer. There is no previously published research on strategies used by test-takers in answering drag-and-drop questions. We tested 476 MTurk participants under five conditions where key design features of mathematics questions were manipulated. Regardless of the design manipulations, participants mostly used one of the two possible systematic response strategies. Using PRIMs cognitive architecture (Taatgen, 2013), we constructed computational cognitive models to simulate the differences between these two strategies. The models were able to capture participants’ reaction time patterns. Our conclusion based on the models is that most participants apply a cognitively less demanding strategy by offloading cognition on action, which is in line with the idea of strategy selection as rational metareasoning (Falk & Griffiths, 2017).