It's All Fun and Games until Someone Annotates: Video Games with a Purpose for Linguistic Annotation

Annotated data is prerequisite for many NLP applications. Acquiring large-scale annotated corpora is a major bottleneck, requiring sig- nificant time and resources. Recent work has proposed turning annotation into a game to increase its appeal and lower its cost; how- ever, current games are largely text-based and closely resemble traditional annotation tasks. We propose a new linguistic annota- tion paradigm that produces annotations from playing graphical video games. The effec- tiveness of this design is demonstrated using two video games: one to create a mapping from WordNet senses to images, and a sec- ond game that performs Word Sense Disam- biguation. Both games produce accurate re- sults. The first game yields annotation qual- ity equal to that of experts and a cost reduc- tion of 73% over equivalent crowdsourcing; the second game provides a 16.3% improve- ment in accuracy over current state-of-the-art sense disambiguation games with WordNet.