Kenneth Regan
Department of Computer Science and Engineering
University at Buffalo
Friday, November 16, 2012
Skill Inference and Chess Cheating Detection from Big Data
We describe a statistical model of move choice at chess. It generates projected probabilities for every move in every position, based only on computer analysis of the positions and fittable parameters that represent a player's skill level and profile. It also generates confidence intervals for statistics such as a player's aggregate error and best-move frequency as judged by computer chess programs, which enable evaluating statistical allegations of cheating that have plagued chess since the 2006 world championship match. Tens of millions of pages of data were used not only to train the model but also to infer what seem to be natural scaling effects of human cognition. Adjusting for these effects improved the model's accuracy on test data.
The general problem this work aims to solve can be called "converting utilities into probabilities" for fallible agents. The data is big enough to demonstrate that simple frequency fitting vastly out-performs maximum-likelihood methods on this application. Real-world results include evidence that human chess skill has improved (contrary to ubiquitous belief in "rating inflation"), that wide variation in performance between amateur and champion level is intrinsic to chess, and that humans perceive utility differences in proportion to the overall benefit rather than absolutely.