Smoking-Bun

I’m still bewildered by the book The Wisdom Of Crowds, the popular science book that renowned playwright Tom Stoppard (no less) called “brilliant and fascinating”. It’s thesis is that “the many are smarter than the few”, and it sets out to prove why ‘groupthink’ can outperform individuals in many situations.

Here’s an insight from the book describing the meticulous laboratory conditions of the Who Wants To Be A Millionaire studio. Contestants ask questions to win a million, but of interest here is the ‘lifelines’ they can us to answer a question they are stuck on, one of which is a call to a colleague who they consider an ‘expert’, and another is a chance to open up the question and see what the whole audience’s answers might be. Can you see where this is going?

…The ‘experts’ did okay, offering the right answer – under pressure – almost 65 percent of the time. But they paled into comparison to the audiences. Those random crowds of people with nothing better to do on a weekday afternoon than sit in a TV studio picked the right answer 91 percent of the time.

There’s a nice blog piece on the matter, which wonders aloud just how and why the group in this case perform so much better than the expert. The point made is that actually the group is as likely to randomly guess the right answer as the wrong one in a 2-way question, or 25% of the time in a 4-way multiple choice format. If all the others chose randomly, it would only take a handful of the group to actually genuinely know the answer for the audience to correctly answer. If 5 of a 100 actually know the answer in a 4-way question, and all the others vote randomly, then the correct answer is likely to be around 30%, while the incorrect answers would be at around 23%. There’s no group wisdom at work there, simply 5% of the audience know the answer and 95% do not – a very reasonable expectation for any general knowledge question.

There’s another issue here – when the audience favor an answer with less than 50% of their vote (in a 4-way question), that is presumably taken as a ‘correct answer’ by these statistics. But in many ways it is not – a minority of the audience actually favor the given answer. 40% of the audience may favor answer A, but 60% think that A is wrong – it’s all about how you express the statistics. If the majority of the audience don’t know the answer, you could just as fairly describe that as the audience simply not knowing. Their best guess is simply the answer they consider the least wrong.

If an audience member doesn’t think they know the answer – do they just guess? Maybe they are encouraged to, but I wouldn’t – if you’re ignorant of the answer it helps the contestant more if you just stay out of it. Far from the book’s thesis about crowds being wise because of the range of intellects on offer, this could simply be a case of members of a crowd being wise enough not to be part of the crowd.

Of course the expert’s experience is very different – under pressure, and without the luxury of the written question in front of them. Their evidence seems to have been collated into right and wrong answers. Of course, the reality on the show is that the individual experts sometimes fail to give an answer at all in the time allotted. On other occasions they have only a vague hunch that the contestant only acts upon if it confirms what they were already thinking. “I don’t know the answer, but I would guess at answer C” – if that proves correct or false, it’s hardly a glowing endorsement for any simplified reading of their performance.

But here’s the biggie – the fact that blows these silly statistics totally out of the water.

Any contestant with half a brain knows that you should only ask the audience a question you think they can answer. Something about popular culture that most people know but maybe you don’t. Or something that you think is general knowledge but you just can’t remember. These, almost by design, tend to be some of the earlier, and generally easier, questions. A good rule of thumb is that the first 5 or so questions are questions that the question-setter expects you to be able to answer – so if you can’t get them, you can be damned sure the audience will know.

If the question is something tougher, that you think is a specific bit of information that only an expert would know, then you phone the expert. Experts get the tougher questions. The way the game plays out is almost always that the audience is used on a fairly easy question, and the expert is used for a harder question. And usually as a last resort – often an expert is called upon in desperation as the last lifeline used.

In conclusion I don’t think the laboratory of the gameshow offers us much insight here. I think experts get the tougher questions. Audiences get the easier ones. Audiences are allowed to offer a vague indicator of the answer, rather than a definitive decision. Experts are under pressure. Audience members get to see the question for a few minutes beforehand. Analysis suggests that a very few audience members who know the answer can totally sway the statistics – this isn’t wisdom, it’s maths.

Renowned playwright Tom Stoppard may be fascinated by the book’s brilliant observations, and I certainly have no reason to doubt the reputation of the man who wrote Rosencrantz and Guildenstern are Dead, one of the most celebrated plays of the last century. But I must say I’m disenchanted by this book’s thesis. But then I am not a crowd, I am one person, and if the many are smarter than the few I am undoubtedly wrong, as the book was a bestseller and Tom Stoppard is a world-renowned playwright.