Although many psychometric tests, like Raven’s Progressive Matrices, are commonly evaluated according to total score, additional variables can lend insight into the underlying cognitive processes. We examine conceptual errors on the Raven’s Standard Progressive Matrices (SPM) test. We present a complete classification of error types on the SPM using a two-kind coding scheme, yielding ≥ 95% inter-rater reliability. We also examine how to extract error data from a computational model, and we present a method for measuring errors through systematic ablation to create a “population” of models whose performance can be examined as a group. We present a preliminary analysis of error patterns on the SPM from typically developing individuals, individuals diagnosed with autism, and a computational model called ASTI. We discuss what the error patterns suggest regarding cognition on the SPM and routes towards improving the ASTI model.
Recent News
New Podcast on Jill Watson and SAMI
Hiring a full-time research scientist and a half-time post-doc
News coverage on Jill Watson: what different sectors can teach us about AI
Congratulations to DILab alumni Mukundan Kuthalam for his recent acceptance to the Computer Science PhD program at Northwestern University!
Congratulations to DILab alumni Varsha Achar for starting her new job at Facebook!