Judges and examiners get laxer with practice

Students are widely judged on their abilities before being allowed to enter top universities. Athletes are assessed on their physical prowess before being awarded medals. And academic papers, like those reported in this section, must run the gauntlet of peer review before being published. In making their determinations, evaluators study that which they are judging in a sequence, one student, athlete or paper after another, and apply standardised criteria. This approach is supposed to afford equal treatment to all. But research just published in Psychological Science by Kieran O’Connor and Amar Cheema of the University of Virginia suggests that it is actually biased in favour of those who are judged late in the process.

Dr O’Connor and Dr Cheema wondered whether making repeated evaluations led judges to feel that their decisions became easier, and if so, whether this increased fluency ultimately led them, unknowingly, to view the evaluation process and evaluations encountered late in a sequence more favourably. To test their idea they looked at judges’ ratings of professional dance competitors across 20 seasons of a television series called “Dancing With The Stars”. They also studied the grades awarded in 1,358 university courses that had been offered by the same lecturer for at least three semesters.

Read the whole story (subscription may be required): The Economist