Bear with me, this will get relevant!
The world of climbing has, somewhat oddly, been riven with controversy over scales. One of the UK's top climbers, James Pearson, made a first ascent of a very hard and dangerous new route which he named “Walk of Life”. So far, so good.
He decided to give the route a grade of E12 which, without getting into the technicalities of the British trad grading system, would make it the hardest climb in the world. Many armchair critics were sceptical about the grade, and the situation got even more heated when Dave MacLeod repeated the route and gave it a grade of E9 (still very hard and dangerous, mind you).
[Image source: http://davemacleod.blogspot.com/]
The poorly-informed and ill-tempered debate that followed in the climbing media (which both Dave and James rose above) focused on whether the grading scale is “broken”, or whether climbers give grades in order to attract publicity/make themselves look hard.
I think this is missing the point completely. The real lesson is that it is impossible for humans to give a completely objective measurement of anything. This is similar to the objection that “one person's 8 is another's 9” or, for that matter, “one person's satisfied is another's very satisfied”.
It's never a good idea to rely too much on a single score given by a single person, but the law of large numbers means that, over a sufficient number of people, average scores will stabilise and be reliable. This is the magic of quantitative surveying—averaging lots of unreliable scores gives us a reliable average.
Interestingly, the few sensible comments about the Wall of Life grading debate draw attention to the fact that climbing grades are always established by consensus. This is simply an informal version of the same phenomenon.