Improving Peer Grading Accuracy in MOOCs: The Effects of Prior Experience, Time Spent on Grading, and Training

Abstract

This paper reports on a study conducted on a course offering of KyotoUx to investigate the effect of training on improving the accuracy of peer assessment. In addition, this study explores the effects of two other variables: the time spent on grading and the prior peer assessment experience on students’ grading accuracy. The results of the analyses indicated that inter-rater agreement was higher between those peer graders who had higher levels of involvement with the training materials. Also, there were significantly higher correlation indices between peer graders more involved with the training materials and the teaching assistant with respect to their grades on the same submission. The time spent on grading and prior assessment experience did not turn out to be major influencing factors on students’ accuracy of grading in this study. We made several suggestions to further optimise the effectiveness of peer assessment in MOOCs and put forth several directions for further research.