Estimating Ability of Programming Skills using IRT based Peer Assessments

Here we analyze data coming from several peer assessment activities, conducted in the framework of a course on basics in Computer Science, for an undergraduate program in Computer Engineering. The peer assessment data are basically the grades given by the peers to the artifacts produced by other peers, while accomplishing a task related to Computer Programming. We analyze the experimental data using an item response theory (IRT) based generalized many-facet Rasch model (GMFRM). Though the peer ratings in five separated sessions of peer assessment were sparse (due to unstable - non mandatory - participation in the activities), the estimated ability and rating characteristics for all participants were extracted. The estimated ability strongly correlates with teacher's rating of the artifacts, and with the final grade obtained by the students in a separate final examination.